Multi-modal key information extraction and summarization method and system based on large language model

Abstract

The invention discloses a multi-modal key information extraction and summarization method and system based on a large language model. The system comprises a preprocessing unit used for preprocessing data of which key information needs to be extracted, and a key information extraction unit used for identifying and extracting named entity and event information through a large language model; the data acquisition unit is used for acquiring data; the parameter setting unit is used for setting various parameters; the core processing unit is used for receiving the data acquired by the data acquisition unit, analyzing and processing the received data and generating a report about the multi-modal key information extraction and summarization system based on the large language model; the summary generation unit is used for summarizing the extracted multi-modal key information according to the report; and the display reminding unit is used for acquiring and displaying the report and performing warning reminding when the report reaches a preset reminding threshold value. According to the scheme, efficient extraction of the document information is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a multimodal key information extraction and summarization method and system based on a large language model. Background Art

[0002] Key information extraction is an important task in the field of natural language processing (NLP), which aims to extract useful information from unstructured text and convert it into structured data for further analysis and application. The development of information extraction technology aims to solve the problem of information overload in large-scale text data processing, providing powerful analysis tools for enterprises, research institutions and government departments. Traditional information extraction methods usually rely on rules and key information matching, which is limited by the structure of the text and the existence of specific key information. This means that the performance of traditional information extraction methods will decline when the document format is irregular or the user question is not clear key information. With the development of large language models, they have achieved remarkable success in natural language processing tasks. However, there are still challenges in the application of information extraction, especially how to effectively combine large language models with text recognition and information extraction techniques to achieve efficient document information extraction. Summary of the invention

[0003] In order to solve the deficiencies in the prior art, the present invention provides a multimodal key information extraction and summarization method and system based on a large language model to solve the technical problem of combining a large language model with text recognition and information extraction technology to achieve efficient document information extraction.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions.

[0005] The present invention firstly discloses a multimodal key information extraction and summarization system based on a large language model, comprising: a preprocessing unit, a key information extraction unit, a data collection unit, a data transmission unit, a parameter setting unit, a core processing unit, a summary generation unit and a display reminder unit;

[0006] The preprocessing unit is used to perform format unification, noise removal and cleaning on the data for which key information needs to be extracted, and to perform operations such as word segmentation, part-of-speech tagging, stop word removal, special character removal, punctuation removal and HTML tag removal on the data;

[0007] The key information extraction unit is used to identify and extract named entities and event information through a large language model, identify semantic relationships between entities, and extract attribute information of entities and events;

[0008] The data collection unit is used to collect the number of keywords, the number of words, the duration and the time data required by the multimodal key information extraction and summarization system based on the large language model;

[0009] The data transmission unit is used to transmit the data collected by the data collection unit to the core processing unit.

[0010] The parameters set by the parameter setting unit are transmitted to the core processing unit and the display reminder unit, and the report generated by the core processing unit is transmitted to the summary generation unit and the display reminder unit;

[0011] The parameter setting unit is used to set various parameters required for analyzing the multimodal key information extraction and summarization system based on the large language model;

[0012] The core processing unit is used to receive the data collected by the data collection unit, analyze and process the received data, and generate a report on the multimodal key information extraction and summary system based on the large language model according to the analysis and processing results;

[0013] The summary generating unit is used to summarize the extracted multimodal key information according to the report on the multimodal key information extraction and summarization system based on the large language model transmitted by the core processing unit;

[0014] The display reminder unit is used to obtain the report generated by the core processing unit on the multimodal key information extraction and summary system based on the large language model, and to display the obtained report, and to issue a warning reminder when the report result reaches a preset reminder threshold.

[0015] The present invention further includes the following preferred embodiments:

[0016] The data collection unit further includes a keyword quantity collection unit, a word quantity collection unit, a duration collection unit and a time collection unit; the keyword quantity collection unit is used to collect various data on the quantity of keywords required by the multimodal key information extraction and summarization system based on a large language model; the word quantity collection unit is used to collect various data on the quantity of words required by the multimodal key information extraction and summarization system based on a large language model; the duration collection module is used to collect the duration data required by the multimodal key information extraction and summarization system based on a large language model; the time collection module is used to collect various data on time required by the multimodal key information extraction and summarization system based on a large language model.

[0017] The parameter setting unit includes a modality weight setting module and a reminder threshold setting module; the modality weight setting module is used to set the weight coefficient of each modality of the key information extracted by the multimodal key information extraction and summary system based on the large language model; the reminder threshold setting module is used to set the reminder threshold required by the multimodal key information extraction and summary system based on the large language model.

[0018] The core processing unit includes a data receiving module, a data analysis module and a report generating module; the data receiving module is used to receive various data required by the multimodal key information extraction and summarization system based on the large language model collected by the data acquisition unit and the weight coefficient of each mode of the key information extracted by the multimodal key information extraction and summarization system based on the large language model set by the parameter setting unit; the data analysis module is used to analyze and process the various data required by the multimodal key information extraction and summarization system based on the large language model received by the data receiving module; the report generating module is used to generate a report on the multimodal key information extraction and summarization system based on the large language model according to the analysis and processing results of the data analysis module.

[0019] Obtain the number of each key information in each mode, the total number of key information in each mode, the number of words that each key information appears for the first time in the data, and the number of words that appear for the last time, and obtain the importance index ω of each key information based on the above data analysis and processing:

[0020]

[0021] Among them, N a is the total amount of information for each modality, n a is the number of each key information in each mode, I a is the weight coefficient of each mode of key information, x is the number of modes in the data, n f is the number of characters when each key information first appears in the data, n l The number of characters that each key information appears in the data for the last time. The importance of each key information is obtained based on its importance index.

[0022] Obtain the number of correctly extracted key information, the total number of key information, the number of all key information that should be extracted, the total number of information in each modality, the request start time, the processing end time, and the processing time, and analyze and process the above data to obtain the performance evaluation index of the multimodal key information extraction and summary system based on the large language model

[0023]

[0024] Among them, RG is the number of key information correctly extracted, ZG is the total number of key information, YG is the number of all key information that should be extracted, and N a is the total amount of information for each mode, x is the number of modes in the data, t S is the request start time, t E is the processing end time, d C is the processing time. Thus, the performance of the multimodal key information extraction and summarization system based on the large language model is obtained according to the performance evaluation index of the multimodal key information extraction and summarization system based on the large language model.

[0025] The display and reminder unit includes a data acquisition module, a report display module and a warning reminder module. The data acquisition module is used to acquire the reminder threshold set by the parameter preset unit and the report of the multimodal key information extraction and summary system based on the large language model generated by the core processing unit; the report display module is used to display the report on the multimodal key information extraction and summary system based on the large language model obtained by the data acquisition module; the warning reminder module is used to issue a warning reminder when the report result on the multimodal key information extraction and summary system based on the large language model obtained by the data acquisition module reaches the reminder threshold set by the parameter preset unit.

[0026] The present invention also discloses a method for extracting and summarizing multimodal key information based on a large language model using the aforementioned multimodal key information extraction and summarization system based on a large language model, comprising:

[0027] Step 1: pre-process the data that needs to be extracted and summarized for key information;

[0028] Step 2: Identify and extract named entities and key event information from the preprocessed data based on the large language model;

[0029] Step 3, collecting data required by the multimodal key information extraction and summarization system based on the large language model, wherein the data includes data on the number of keywords, data on the number of words, data on duration, and data on time;

[0030] Step 4, setting the parameters required by the multimodal key information extraction and summarization system based on the large language model, wherein the parameters include a weight coefficient of each mode of the key information and a reminder threshold;

[0031] Step 5, analyzing and processing the parameters and data and generating a report on a multimodal key information extraction and summary system based on a large language model;

[0032] Step 6: Summarize the report generated by the extracted key information combined with the analysis and processing, and display the report on the multimodal key information extraction and summary system based on the large language model. At the same time, issue a warning reminder when the report result reaches a preset reminder threshold.

[0033] Accordingly, the present application also discloses a terminal, including a processor and a storage medium;

[0034] The storage medium is used to store instructions;

[0035] The processor is used to operate according to the instructions to execute the steps of the aforementioned multimodal key information extraction and summarization method based on a large language model.

[0036] Correspondingly, the present application also discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the aforementioned multimodal key information extraction and summarization method based on a large language model are implemented.

[0037] The beneficial effect of the present invention is that, compared with the prior art, the present invention provides a multimodal key information extraction and summarization method and system based on a large language model, which realizes efficient key information extraction by combining a large language model with a traditional algorithm, thereby improving the efficiency of key information extraction, and at the same time, by real-time monitoring and real-time monitoring of the processing performance of the multimodal key information extraction and summarization system based on the large language model, a warning reminder is issued when the system performance reaches the reminder threshold, so that the performance problems of the multimodal key information extraction and summarization system based on the large language model can be discovered in time, and timely processed to improve the performance of the multimodal key information extraction and summarization system based on the large language model, and the number of each key information in each mode, the total number of key information in each mode, the number of words that each key information appears for the first time in the data, and the number of words that appear for the last time are obtained during the analysis and processing, and the importance index of each key information is obtained according to the above data analysis and processing, the larger the importance index is, the higher the importance of the key information is, and higher attention is given when summarizing, and vice versa, the lower the importance of the key information is, and lower attention is given when summarizing. At the same time, the number of correctly extracted key information, the total number of key information, the number of all key information that should be extracted, the total amount of information in each modality, the request start time, the processing end time, and the processing time can be analyzed and processed to obtain the performance evaluation index of the multimodal key information extraction and summary system based on the large language model. The larger the performance evaluation index, the better the performance of the multimodal key information extraction and summary system based on the large language model. Conversely, the worse the performance of the multimodal key information extraction and summary system based on the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1It is a structural diagram of the multimodal key information extraction and summarization system based on a large language model in the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clear, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0040] The embodiments described in this application are only some embodiments of the present invention, not all embodiments. Based on the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the protection scope of the present invention.

[0041] In view of the shortcomings of the prior art, the present invention proposes a multimodal key information extraction and summarization method and system based on a large language model, which realizes efficient key information extraction by combining the large language model with the traditional algorithm, thereby improving the efficiency of key information extraction. At the same time, the processing performance of the multimodal key information extraction and summarization system such as video, voice, picture, and text is monitored in real time, and a warning reminder is issued when the system performance reaches the reminder threshold, and timely processing is performed to improve the performance of the system.

[0042] See also Figure 1 As shown, the multimodal key information extraction and summarization system based on the large language model disclosed in the present invention includes a preprocessing unit, a key information extraction unit, a data acquisition unit, a data transmission unit, a parameter setting unit, a core processing unit, a summary generation unit and a display reminder unit.

[0043] The preprocessing unit is used to perform format unification processing, noise removal and cleaning on the data from which key information needs to be extracted, and to perform operations of word segmentation, part-of-speech tagging, stop word removal, special character removal, punctuation removal and HTML tag removal on the data.

[0044] The key information extraction unit is used to identify and extract named entities and event information through a large language model, identify semantic relationships between entities, and extract attribute information of entities and events.

[0045] The data collection unit is used to collect the number of keywords, number of words, duration and time data required by the multimodal key information extraction and summarization system based on the large language model.

[0046] In a further embodiment, the data collection unit includes: a keyword quantity collection unit, a word quantity collection unit, a duration collection unit and a time collection unit. The keyword quantity collection unit is used to collect various data on the quantity of keywords required for the multimodal key information extraction and summarization system based on a large language model. The word quantity collection unit is used to collect various data on the quantity of words required for the multimodal key information extraction and summarization system based on a large language model. The duration collection module is used to collect the duration data required for the multimodal key information extraction and summarization system based on a large language model. The time collection module is used to collect various data on time required for the multimodal key information extraction and summarization system based on a large language model.

[0047] The data transmission unit is used to transmit the data collected by the data collection unit to the core processing unit, and transmit the parameters set by the parameter setting unit to the core processing unit and the display reminder unit, and at the same time transmit the report generated by the core processing unit to the summary generation unit and the display reminder unit.

[0048] The parameter setting unit is used to set various parameters required for analyzing the multimodal key information extraction and summarization system based on the large language model.

[0049] In a further embodiment, the parameter setting unit includes: each modality weight setting module and a reminder threshold setting module. The each modality weight setting module is used to set the weight coefficient of each modality of the key information extracted by the multimodal key information extraction and summarization system based on the large language model. The reminder threshold setting module is used to set the reminder threshold required by the multimodal key information extraction and summarization system based on the large language model.

[0050] The core processing unit is used to receive the data collected by the data collection unit, analyze and process the received data, and generate a report on the multimodal key information extraction and summary system based on the large language model according to the analysis and processing results.

[0051] In a further embodiment, the core processing unit includes: a data receiving module, a data analysis module and a report generation module. The data receiving module is used to receive the various data required by the multimodal key information extraction and summary system based on the large language model collected by the data collection unit and the weight coefficient of each mode of the key information extracted by the multimodal key information extraction and summary system based on the large language model set by the parameter setting unit. The data analysis module is used to analyze and process the various data required by the multimodal key information extraction and summary system based on the large language model received by the data receiving module. The report generation module is used to generate a report on the multimodal key information extraction and summary system based on the large language model according to the analysis and processing results of the data analysis module.

[0052] In a further embodiment, when the data analysis module performs analysis and processing, the number of each key information in each mode, the total number of key information in each mode, the number of words that each key information appears for the first time in the data, and the number of words that appear for the last time are obtained, and the importance index ω of each key information is obtained according to the above data analysis and processing:

[0053]

[0054] Among them, N a is the total amount of information for each modality, n a is the number of each key information in each mode, I a is the weight coefficient of each mode of key information, x is the number of modes in the data, n f is the number of characters when each key information first appears in the data, n l The number of characters that each key information appears in the data for the last time. The importance of each key information is obtained based on its importance index. The larger the importance index, the more important the key information is, and more attention should be paid to it when summarizing it. Conversely, the lower the importance of the key information, the less attention should be paid to it when summarizing it.

[0055] In a further embodiment, the data analysis module obtains the number of correctly extracted key information, the total number of key information, the number of all key information to be extracted, the total number of information in each modality, the request start time, the processing end time and the processing time during the analysis and processing, and analyzes and processes the above data to obtain the performance evaluation index of the multimodal key information extraction and summary system based on the large language model

[0056]

[0057] Among them, RG is the number of key information correctly extracted, ZG is the total number of key information, YG is the number of all key information that should be extracted, and N a is the total amount of information for each mode, x is the number of modes in the data, t S is the request start time, t E is the processing end time, d C The processing time is as follows. Thus, the performance evaluation index of the multimodal key information extraction and summarization system based on the large language model is used to determine the performance of the multimodal key information extraction and summarization system based on the large language model. The larger the performance evaluation index, the better the performance of the multimodal key information extraction and summarization system based on the large language model. Conversely, the worse the performance of the multimodal key information extraction and summarization system based on the large language model.

[0058] The summary generating unit is used to summarize the extracted multimodal key information according to the report on the multimodal key information extraction and summarization system based on the large language model transmitted by the core processing unit.

[0059] The display reminder unit is used to obtain the report on the multimodal key information extraction and summary system based on the large language model generated by the core processing unit, and display the obtained report. At the same time, when the report result reaches a preset reminder threshold, a warning reminder is issued so that the operator can promptly discover the problems of the multimodal key information extraction and summary system based on the large language model, and process them to improve the performance of the multimodal key information extraction and summary system based on the large language model.

[0060] In a further embodiment, the display reminder unit includes a data acquisition module, a report display module and a warning reminder module. The data acquisition module is used to acquire the reminder threshold set by the parameter preset unit and the report of the multimodal key information extraction and summary system based on the large language model generated by the core processing unit. The report display module is used to display the report on the multimodal key information extraction and summary system based on the large language model obtained by the data acquisition module. The warning reminder module is used to issue a warning reminder when the report result on the multimodal key information extraction and summary system based on the large language model obtained by the data acquisition module reaches the reminder threshold set by the parameter preset unit.

[0061] The present invention also discloses a multimodal key information extraction and summarization method based on a large language model based on the aforementioned multimodal key information extraction and summarization system based on a large language model, comprising:

[0062] Step 1: Preprocess the data that needs to be summarized and extracted for key information.

[0063] Specifically, the preprocessing unit unifies the format of the data for extracting key information, performs denoising and cleaning, and performs word segmentation, part-of-speech tagging, removes stop words, removes special characters, removes punctuation marks, and removes HTML tags on the data.

[0064] Step 2: Identify and extract named entities and key event information from the preprocessed data based on the large language model.

[0065] The key information extraction unit recognizes and extracts named entities and event information based on a large language model, identifies the semantic relationship between entities, and extracts attribute information of entities and events.

[0066] Step 3, collecting data required by the multimodal key information extraction and summarization system based on the large language model, wherein the data includes data on the number of keywords, data on the number of words, data on duration, and data on time;

[0067] Through the keyword quantity collection unit, the word quantity collection unit, the duration collection unit and the time collection unit of the data collection unit, various data on the number of keywords required by the multimodal key information extraction and summary system based on a large language model, various data on the number of words required by the multimodal key information extraction and summary system based on a large language model, various data on duration required by the multimodal key information extraction and summary system based on a large language model, and various data on time required by the multimodal key information extraction and summary system based on a large language model are collected.

[0068] Step 4, setting the parameters required by the multimodal key information extraction and summarization system based on the large language model, wherein the parameters include a weight coefficient of each mode of the key information and a reminder threshold;

[0069] The weight coefficient of each modality of the key information extracted by the multimodal key information extraction and summary system based on the large language model is set through the modal weight setting modules of the parameter setting unit, and the reminder threshold required by the multimodal key information extraction and summary system based on the large language model is set through the reminder threshold setting module.

[0070] Step 5: Analyze and process the parameters and data and generate a report on the multimodal key information extraction and summary system based on the large language model.

[0071] When the data analysis module performs analysis and processing, it can obtain the number of each key information in each mode, the total number of key information in each mode, the number of words when each key information first appears in the data, and the number of words when it last appears, and obtain the importance index of each key information based on the above data analysis and processing, so as to derive the importance of each key information based on the importance index of each key information. The larger the importance index, the higher the importance of the key information, and more attention will be given to it when summarizing. Conversely, the lower the importance of the key information, the lower the attention will be given to it when summarizing.

[0072] At the same time, during the analysis and processing, the data analysis module can analyze and process the number of correctly extracted key information, the total number of key information, the number of all key information that should be extracted, the total number of information in each modality, the request start time, the processing end time, and the processing time, and analyze and process the above data to obtain the performance evaluation index of the multimodal key information extraction and summary system based on the large language model, thereby deriving the performance of the multimodal key information extraction and summary system based on the large language model according to the performance evaluation index of the multimodal key information extraction and summary system based on the large language model. The larger the performance evaluation index, the better the performance of the multimodal key information extraction and summary system based on the large language model, and vice versa.

[0073] Step 6: Summarize the report generated by the extracted key information combined with the analysis and processing, and display the report on the multimodal key information extraction and summary system based on the large language model. At the same time, issue a warning reminder when the report result reaches a preset reminder threshold.

[0074] Specifically, the extracted multimodal key information is summarized by the summary generation unit according to the report on the multimodal key information extraction and summary system based on the large language model transmitted by the core processing unit, and the report on the multimodal key information extraction and summary system based on the large language model is displayed by the report display module of the display reminder unit. At the same time, when the report result reaches a preset reminder threshold, a warning reminder module is used to issue a warning reminder.

[0075] The beneficial effect of the present invention is that, compared with the prior art, the present invention provides a multimodal key information extraction and summarization method and system based on a large language model, which realizes efficient key information extraction by combining a large language model with a traditional algorithm, thereby improving the efficiency of key information extraction, and at the same time, by real-time monitoring and real-time monitoring of the processing performance of the multimodal key information extraction and summarization system based on the large language model, a warning reminder is issued when the system performance reaches the reminder threshold, so that the performance problems of the multimodal key information extraction and summarization system based on the large language model can be discovered in time, and timely processed to improve the performance of the multimodal key information extraction and summarization system based on the large language model, and the number of each key information in each mode, the total number of key information in each mode, the number of words that each key information appears for the first time in the data, and the number of words that appear for the last time are obtained during the analysis and processing, and the importance index of each key information is obtained according to the above data analysis and processing, the larger the importance index is, the higher the importance of the key information is, and higher attention is given when summarizing, and vice versa, the lower the importance of the key information is, and lower attention is given when summarizing. At the same time, the number of correctly extracted key information, the total number of key information, the number of all key information that should be extracted, the total amount of information in each modality, the request start time, the processing end time, and the processing time can be analyzed and processed to obtain the performance evaluation index of the multimodal key information extraction and summary system based on the large language model. The larger the performance evaluation index, the better the performance of the multimodal key information extraction and summary system based on the large language model. Conversely, the worse the performance of the multimodal key information extraction and summary system based on the large language model.

[0076] Based on the spirit of the present invention, those skilled in the art can easily think that a computer program product can be obtained based on the aforementioned multimodal key information extraction and summarization method based on a large language model. The computer program product may include a computer-readable storage medium, which carries computer-readable program instructions for enabling a processor to implement various aspects of the present disclosure. That is, the present application also includes a terminal, including a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the aforementioned multimodal key information extraction and summarization method based on a large language model.

[0077] Computer readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device. Computer readable storage medium can be, for example, - but not limited to - electrical storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device or any suitable combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, for example, punch card or groove protrusion structure with instructions stored thereon and any suitable combination of the above. Computer readable storage medium used here is not interpreted as instantaneous signal itself, such as radio wave or other free propagating electromagnetic wave, electromagnetic wave propagated by waveguide or other transmission medium (for example, light pulse by optical fiber cable) or electrical signal transmitted by wire.

[0078] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0079] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages-such as Smalltalk, C++, etc., and conventional procedural programming languages-such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network-including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized by utilizing the state information of a computer-readable program instruction, and the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multimodal key information extraction and summarization system based on a large language model, characterized in that: It includes a pre-processing unit, a key information extraction unit, a data collection unit, a data transmission unit, a parameter setting unit, a core processing unit, a summary generation unit and a display reminder unit; The preprocessing unit is used to perform format unification, noise removal and cleaning on the data for which key information needs to be extracted, and to perform operations such as word segmentation, part-of-speech tagging, stop word removal, special character removal, punctuation removal and HTML tag removal on the data; The key information extraction unit is used to identify and extract named entities and event information through a large language model, identify semantic relationships between entities, and extract attribute information of entities and events; The data collection unit is used to collect the number of keywords, the number of words, the duration and the time data required by the multimodal key information extraction and summarization system based on the large language model; The data transmission unit is used to transmit the data collected by the data collection unit to the core processing unit. The parameters set by the parameter setting unit are transmitted to the core processing unit and the display reminder unit, and the report generated by the core processing unit is transmitted to the summary generation unit and the display reminder unit; The parameter setting unit is used to set various parameters required for analyzing the multimodal key information extraction and summarization system based on the large language model; The core processing unit is used to receive the data collected by the data collection unit, analyze and process the received data, and generate a report on the multimodal key information extraction and summary system based on the large language model according to the analysis and processing results; The summary generating unit is used to summarize the extracted multimodal key information according to the report on the multimodal key information extraction and summarization system based on the large language model transmitted by the core processing unit; The display reminder unit is used to obtain the report generated by the core processing unit on the multimodal key information extraction and summary system based on the large language model, and to display the obtained report, and to issue a warning reminder when the report result reaches a preset reminder threshold.

2. The multimodal key information extraction and summarization system based on a large language model according to claim 1 is characterized in that: The data collection unit further includes a keyword quantity collection unit, a word quantity collection unit, a duration collection unit, and a time collection unit; the keyword quantity collection unit is used to collect various data on the number of keywords required by the multimodal key information extraction and summarization system based on the large language model; The character quantity collection unit is used to collect various data on the character quantity required by the multimodal key information extraction and summarization system based on the large language model; The duration collection module is used to collect the duration data required by the multimodal key information extraction and summarization system based on the large language model; The time collection module is used to collect various time-related data required by the multimodal key information extraction and summarization system based on the large language model.

3. The multimodal key information extraction and summarization system based on a large language model according to claim 2 is characterized in that: The parameter setting unit includes a modality weight setting module and a reminder threshold setting module; the modality weight setting module is used to set the weight coefficient of each modality of the key information extracted by the multimodal key information extraction and summarization system based on the large language model; The reminder threshold setting module is used to set the reminder threshold required by the multimodal key information extraction and summarization system based on the large language model.

4. The multimodal key information extraction and summarization system based on a large language model according to claim 3 is characterized in that: The core processing unit includes a data receiving module, a data analysis module and a report generating module; the data receiving module is used to receive various data required by the multimodal key information extraction and summarization system based on the large language model collected by the data collection unit and the weight coefficient of each mode of the key information extracted by the multimodal key information extraction and summarization system based on the large language model set by the parameter setting unit; the data analysis module is used to analyze and process various data required by the multimodal key information extraction and summarization system based on the large language model received by the data receiving module; The report generation module is used to generate a report on the multimodal key information extraction and summary system based on the large language model according to the analysis and processing results of the data analysis module.

5. The multimodal key information extraction and summarization system based on a large language model according to claim 4 is characterized in that: The data analysis module is further used to obtain the number of each key information in each mode, the total number of key information in each mode, the number of words that each key information appears for the first time in the data, and the number of words that appear for the last time, and obtain the importance index ω of each key information according to the above data analysis and processing: Among them, N a is the total amount of information for each modality, n a is the number of each key information in each mode, I a is the weight coefficient of each mode of key information, x is the number of modes in the data, n f is the number of characters when each key information first appears in the data, n l The number of characters that each key information appears for the last time in the data; the importance of each key information is obtained based on its importance index.

6. The multimodal key information extraction and summarization system based on a large language model according to claim 5 is characterized in that: The data analysis module is further used to obtain the number of correctly extracted key information, the total number of key information, the number of all key information that should be extracted, the total number of information in each modality, the request start time, the processing end time and the processing time, and analyze and process the above data to obtain the performance evaluation index φ of the multimodal key information extraction and summary system based on the large language model: Among them, RG is the number of key information correctly extracted, ZG is the total number of key information, YG is the number of all key information that should be extracted, and N a is the total amount of information for each mode, x is the number of modes in the data, t S is the request start time, t E is the processing end time, d C is the processing time; the performance of the multimodal key information extraction and summarization system based on a large language model is obtained according to the performance evaluation index of the multimodal key information extraction and summarization system based on a large language model.

7. The multimodal key information extraction and summarization system based on a large language model according to claim 6 is characterized in that: The display and reminder unit includes a data acquisition module, a report display module and a warning reminder module. The data acquisition module is used to acquire the reminder threshold set by the parameter preset unit and the report of the multimodal key information extraction and summary system based on the large language model generated by the core processing unit; the report display module is used to display the report on the multimodal key information extraction and summary system based on the large language model acquired by the data acquisition module; The warning reminder module is used to issue a warning reminder when the report result of the multimodal key information extraction and summary system based on the large language model obtained by the data acquisition module reaches the reminder threshold set by the parameter preset unit.

8. A method for extracting and summarizing multimodal key information based on a large language model based on the multimodal key information extraction and summarization system based on a large language model according to any one of claims 1 to 7, characterized in that: include: Step 1: pre-process the data that needs to be extracted and summarized for key information; Step 2: Identify and extract named entities and key event information from the preprocessed data based on the large language model; Step 3, collecting data required by the multimodal key information extraction and summarization system based on the large language model, wherein the data includes data on the number of keywords, data on the number of words, data on duration, and data on time; Step 4, setting the parameters required by the multimodal key information extraction and summarization system based on the large language model, wherein the parameters include a weight coefficient of each mode of the key information and a reminder threshold; Step 5, analyzing and processing the parameters and data and generating a report on a multimodal key information extraction and summary system based on a large language model; Step 6: Summarize the report generated by the extracted key information combined with the analysis and processing, and display the report on the multimodal key information extraction and summary system based on the large language model. At the same time, issue a warning reminder when the report result reaches a preset reminder threshold.

9. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the multimodal key information extraction and summarization method based on a large language model according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the multimodal key information extraction and summarization method based on a large language model described in any one of claims 1 to 7 are implemented.

Images