Data content extraction method, system and equipment based on cue word and medium

Abstract

The invention relates to the technical field of multi-modal data processing, in particular to a data content extraction method, system and equipment based on cue words and a medium. The invention provides a cue word-based multi-modal data content extraction method, system and device and a medium, and the method comprises the steps: configuring an operation environment, designing and optimizing cue words, extracting multi-modal data and generating labels, storing and managing the labels, calculating the multi-modal data by using a large language model, obtaining original content description data, and extracting the original content description data by using the cue word-based multi-modal data. Analyzing the original data, storing the original data into a relational database, and indexing the original data according to classification; and batch data processing. The method aims at solving the problem that in the prior art, content extraction of multi-modal data cannot be conducted through cue words.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and in particular to a data content extraction method, system, device and medium based on prompt words. Background Art

[0002] With the rapid development of digital media platforms, user-generated content presents multimodal characteristics such as video, image, audio and text. Studying Internet user behavior and communication patterns has become an important topic in the fields of social sciences and journalism and communication. However, existing technologies have the following problems when analyzing these data: 1. High complexity in multimodal data processing: The characteristics of video, image, audio and text data are significantly different. Traditional methods usually require separate extraction and analysis algorithms for each modality, resulting in high technical barriers and difficulty in achieving unified processing of multimodal data.

[0003] 2. Lack of flexibility in data labeling: Existing systems usually limit data labeling to predefined dimensions and lack a flexible label generation mechanism for different analysis objectives, which limits the deep mining and application of data.

[0004] 3. Challenges brought by data scale: Faced with the massive amount of user content on digital media platforms, existing technologies are difficult to efficiently process and analyze multimodal data, especially when building large-scale, multi-dimensional tag libraries.

[0005] The Chinese invention patent application with publication number CN106021234A disclosed a label extraction method and system on October 12, 2016, which belongs to the field of language recognition technology and can achieve relatively accurate label extraction. The label extraction method includes: obtaining comments from a database; marking the part of speech of words in the comments; extracting keywords in each comment according to the part of speech tagging results; and generating phrases containing the keywords based on the extracted keywords. The embodiment of the patent application can be applied to the label extraction of music, commodities and other things with high complexity, but it is mainly for the extraction of comment text, which has great limitations and fails to achieve unified processing and analysis of multimodal data such as video, audio, and images.

[0006] Therefore, how to design an efficient and intelligent multimodal data extraction and labeling analysis method to meet the structured needs of multimodal data in the fields of social sciences and journalism and communication, while lowering the technical threshold of data analysis, has become an urgent problem to be solved. Summary of the invention

[0007] The purpose of the present invention is to overcome the deficiencies of the above-mentioned prior art and to provide a method, system, device and medium for extracting content from multimodal data based on prompt words, which aims to solve the problem in the prior art that content extraction from multimodal data cannot be performed through prompt words.

[0008] To achieve the above objectives, this application proposes the following solutions: In a first aspect, the present invention proposes a method for extracting content from multimodal data based on prompt words, the method comprising: S1, configure the operating environment and provide the basis for extracting the prompt word content; S2, design and optimize prompt words, including designing general prompt word templates, theme customization of prompt words, and loop debugging and optimization of prompt words; S3, multimodal data extraction and label generation, including extracting content labels for videos, images, audio, and text; S4, tag storage and management, uses a large language model to calculate multimodal data to obtain original content description data, parses the original data, stores it in a relational database, and indexes it by category; S5, batch data processing, after completing the calculation of a single prompt word and a single video, image, audio and text, based on the prompt word optimized in step S2, according to the above steps S3 and S4, the data to be processed is looped, and according to the results of each group of data output, multiple label data are stored in the relational database in turn, and indexed by category.

[0009] Preferably, the step of configuring the operating environment in S1 includes: 1) Prepare a high-performance computer host for deploying a large language model environment; 2) Build a visual recognition environment (including videos and images) based on an open source multimodal large model (such as MiniCPM-2.6-V), and provide an API interface through coding; 3) Build a speech recognition environment based on open source speech models (such as Whisper-large) and provide API interfaces through coding; 4) Build a large model text reasoning environment based on open source text large language models (such as DeepSeek) and provide API interfaces through coding; 5) Develop a program that integrates the API interfaces of the above three steps and provides an interface for user operations.

[0010] Preferably, the step of designing and optimizing prompt words in S2 includes: 1) General prompt word template: design general prompt words suitable for basic label extraction of multimodal data; 2) Theme-customized prompt words: Based on the dimensions of cognition, emotion, narrative, atmosphere, and value judgment analysis, the prompt words are optimized to analyze these dimensions and support data labeling of the above dimensions; 3) Cycle debugging and optimization: perform multiple rounds of optimization on the prompt words through the prompt word debugging interface until the analysis requirements are met.

[0011] Preferably, the steps of multimodal data extraction and label generation in S3 include: 1) Video processing: The optimized prompt words in step S2 are passed to the multimodal big model through the interface, and a video to be analyzed is uploaded. The program will parse the key frames of the video, and submit the prompt words and key frame image data to the big model. The video content is extracted according to the output dimensions specified in the prompt words, and structured tags are generated in JSON format; the specified output dimensions are cognitive, emotional, narrative, atmosphere, and value judgment analysis dimensions, and the prompt words are optimized to parse these dimensions; 2) Image processing: The optimized prompt words in step S2 are passed to the multimodal big model through the interface, and an image file to be analyzed is uploaded. The program submits the prompt words and image data to the big model, extracts the image content according to the output dimension specified in the prompt words, and generates structured tags in JSON format; 3) Audio processing: Use the speech recognition model to transcribe the audio content and extract the speech information in the audio. For music audio, we use the music analysis model to extract the audio pitch, intensity, rhythm, spectrum and other information to form structured text content, and store the structured text content. 4) Text analysis: Calculate the text content based on the big language model, provide the structured text content output in step 3) to the text big language model, and extract content tags according to different dimensions based on the given prompt words, and generate structured tag data. The dimensions are cognitive, emotional, narrative, atmosphere, and value judgment analysis dimensions.

[0012] Preferably, the specific method of label storage and management in step S4 is to obtain the original label data through the model calculation in step S3, and we parse the original data to obtain the key-value structure in the cognitive, emotional, narrative, atmosphere, and value judgment analysis dimensions to form structured data. The structured data is the required label data, and then the label data is stored in a relational database and indexed by category.

[0013] In a second aspect, the present invention proposes a multimodal data content extraction system based on prompt words, the system comprising: The prompt word design and optimization module can continuously clarify the key information that needs to be extracted and improve the accuracy of large model recognition; Multimodal data extraction and label generation module, which can analyze, extract and generate labels for video, image, audio and text data; The tag storage and management module uses a large language model to calculate multimodal data to obtain original content description data and tag original data, parses the original content description data, obtains the key-value structure in the cognitive, emotional, narrative, atmosphere, and value judgment analysis dimensions, and thus forms structured data. The structured data is the required tag data, and then the tag data is stored in a relational database and indexed by category. The batch data processing module, based on the optimized prompt words in the prompt word design and optimization module, circulates the data to be processed according to the above multimodal data extraction and label generation module and the label storage and management module, and stores multiple label data in the relational database in turn according to the results of each group of data output, and indexes them by category.

[0014] Furthermore, the prompt word design and optimization module includes a general prompt word template module, a theme customized prompt word module and a loop debugging and optimization module; The general prompt word template module is used to design general prompt words, which is suitable for basic label extraction of multimodal data; Theme-customized prompt word module optimizes prompt words to analyze the dimensions of cognition, emotion, narrative, atmosphere, and value judgment, and supports data labeling of the above dimensions; The loop debugging and optimization module performs multiple rounds of optimization on the prompt words through the prompt word debugging interface until the analysis requirements are met.

[0015] Furthermore, the multimodal data extraction and label generation module includes a video processing module, an image processing module, an audio processing module and a text analysis module; The video processing module passes the optimized prompt words in the prompt word design and optimization module to the multimodal big model through the interface, and uploads a video to be analyzed. The program will parse the key frames of the video and submit the prompt words and key frame image data to the big model. It will extract the video content according to the output dimension specified in the prompt words and generate structured tags in JSON format. The image processing module passes the optimized prompt words in the prompt word design and optimization module to the multimodal big model through the interface, and uploads a picture file to be analyzed. The program submits the prompt words and image data to the big model, extracts the image content according to the output dimension specified in the prompt words, and generates structured tags in JSON format; The audio processing module uses a speech recognition model to transcribe the audio content and extract the speech information in the audio. For music audio, we use a music analysis model to extract the audio pitch, intensity, rhythm, spectrum and other information to form structured text content, and store the structured text content. The text analysis module calculates the text content based on the large language model, provides the structured text content output from the audio processing module to the text large language model, and extracts content tags according to different dimensions based on the given prompt words, and generates structured tag data.

[0016] In a third aspect, the present invention proposes a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0017] In a fourth aspect, the present invention proposes a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described in the first aspect when executed by a processor.

[0018] Compared with the prior art, the method, system, device and medium for extracting data content based on prompt words provided by the present invention have the following beneficial effects: 1. Lowering the technical threshold: By adjusting and optimizing the prompt words, the video, image, audio, and text data can be accurately extracted and multi-dimensionally labeled for analysis without traditional coding, which greatly reduces the technical difficulty of multimodal analysis and makes it easier for researchers and corporate users in the fields of social sciences and journalism and communication to get started. It is suitable for wide application in the fields of social sciences and journalism and communication.

[0019] 2. Support analysis of any dimension: The present invention can customize data analysis dimensions through prompt words to meet the user's cognitive, emotional, narrative, atmosphere, and value judgment analysis dimensions in various scenarios such as communication law research, user behavior, picture style, related topics and description extraction, optimize prompt words to parse these dimensions, support data labeling of the above dimensions, and greatly improve the flexibility and practicality of the analysis.

[0020] 3. Improve the efficiency and accuracy of label extraction: Based on the multi-modal large model, the data is processed uniformly, which can efficiently generate a structured label library without complex feature engineering, and achieve seamless integration of data of different modalities. Through the prompt word optimization mechanism, it ensures that the label extraction results are highly relevant to the analysis topic.

[0021] 4. Optimize large-scale data processing: Support batch processing of large-scale data on social media platforms, significantly improve processing efficiency through multi-threading and automation technology, and is suitable for high-frequency, real-time data analysis scenarios.

[0022] 5. Enhance data availability: Build a unified multimodal tag library to facilitate data storage, retrieval, and reuse, providing high-quality basic data support for subsequent social science and journalism and communication research.

[0023] 6. Wide application scenarios: This technology is applicable to social sciences, news dissemination, government propaganda, medical industry, cultural tourism, advertising and public relations, marketing and other fields. It is aimed at multimodal content extraction and labeling analysis of user behavior data on social media platforms. It can also be expanded to practical application scenarios such as network communication and correlation analysis.

[0024] 7. Empowering scientific research and industry: Through intelligent and automated analysis methods, it promotes technological innovation in the fields of social sciences and journalism and communication, while providing data support for media platforms, government agencies, and corporate users. It has significant academic value and industrialization potential.

[0025] The features and advantages of the present invention will be described in detail through embodiments in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of a method for extracting multimodal data content based on prompt words according to an embodiment of the present invention.

[0027] Figure 2 It is a flowchart of the steps of configuring the operating environment in S1 of the present invention.

[0028] Figure 3 It is a flowchart of the steps of designing and optimizing prompt words in S2 of the present invention.

[0029] Figure 4 It is a flowchart of the steps of multimodal data extraction and label generation in S3 of the present invention.

[0030] Figure 5 It is a module schematic diagram of the system of the present invention.

[0031] Figure 6 Examples of common prompt words.

[0032] Figure 7 Upload images / videos for analysis and get the raw data interface.

[0033] Figure 8 It is the interface of the label table data page formed after the original data is parsed by labeling.

[0034] Fig. 9 It is a speech-to-text interface.

[0035] Fig.10 This is an example of speech tag extraction results. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below through the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of the present invention. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0037] The present invention is mainly used in the fields of social sciences, news dissemination, government propaganda, medical industry, cultural tourism, advertising and public relations, marketing, etc. It is a multimodal content extraction and labeling analysis technology for user behavior data on social media platforms. By adjusting and optimizing the prompt words, the multimodal element extraction and labeling process of video, image, audio and text data is simplified, supporting flexible analysis of data in any dimension. Unlike traditional methods, the present invention does not require complex coding and separate algorithms for a certain mode of video, image, audio, and text. It only needs to optimize the prompt words to achieve efficient and accurate content extraction and label generation, greatly reducing the technical threshold of multimodal data analysis. This method provides an intelligent and automated solution for the fields of social sciences, news dissemination, government propaganda, medical industry, cultural tourism, advertising and public relations, and marketing. It can quickly build a structured tag library and promote the development of applications such as user behavior analysis, communication model research, and information monitoring.

[0038] See also Figure 1-4 Based on the above situation, an embodiment of the present invention provides a method for extracting multimodal data content based on prompt words, the method comprising: S1, configure the operating environment and provide the basis for extracting the prompt word content; S2, design and optimize prompt words, including designing general prompt word templates, customizing prompt word themes, and looping debugging and optimizing theme words; S3, multimodal data extraction and label generation, including video, image, audio and text content label extraction; S4, tag storage and management, uses a large language model to calculate multimodal data to obtain original content description data, parses the original data, stores it in a relational database, and indexes it by category; S5, batch data processing, after completing the calculation of a single prompt word and a single video, image, voice, or text, based on the prompt word optimized in step S2, according to the above steps S3 and S4, the data to be processed is cyclically run, and according to the results of each group of data output, multiple label data are sequentially stored in a relational database (such as MySQL) and indexed by category.

[0039] Furthermore, the present invention realizes efficient analysis and labeling of multimodal data of video, image, audio and text by adjusting prompt words. In this design, the following environment needs to be configured in advance (the following steps are the operating environment configuration, which is the basis for the subsequent realization of prompt words to complete content extraction): S1 is a prerequisite. The steps for configuring the operating environment in S1 include: 1) Prepare a high-performance computer host for deploying a large language model environment; 2) Build a visual recognition environment (including videos and images) based on an open source multimodal large model (such as MiniCPM-2.6-V), and provide an API interface through coding; 3) Build a speech recognition environment based on open source speech models (such as Whisper-large) and provide API interfaces through coding; 4) Build a large model text reasoning environment based on open source text large language models (such as DeepSeek) and provide API interfaces through coding; 5) Develop a program that integrates the API interfaces of the above three steps and provides a page for user operations.

[0040] The operating environment can be configured through the above five steps. Only with a good operating environment can the solution of the present invention be better completed.

[0041] Furthermore, the steps of designing and optimizing prompt words in S2 include: 1) General prompt word template: design general prompt words suitable for basic label extraction of multimodal data, including actions, scenes, characters, texts, etc. 2) Theme-customized prompt words: Based on the dimensions of cognition, emotion, narrative, atmosphere, and value judgment analysis, the prompt words are optimized to analyze these dimensions and support data labeling of the above dimensions; 3) Cycle debugging and optimization: perform multiple rounds of optimization on the prompt words through the prompt word debugging interface until the analysis requirements are met.

[0042] This article uses a specific case to illustrate the steps of S2 design and optimization of prompt words. For example, the following are prompt words that have been optimized in three steps. It is necessary to extract cartoon elements that appear in the video. The first step roughly gives the dimensions that need to be extracted, the second step clarifies the content that needs to be extracted, and the third step continues to add descriptions and definitions of the extracted content. After these three steps of optimization, the key information that needs to be extracted is continuously clarified, and the accuracy of recognition through the large model is also continuously improved.

[0043] 1) Please extract the cartoon elements used in the image and output them as standard JSON format data.

[0044] 2) "Please extract the cartoon image, cartoon scene description, cartoon style, cartoon background color, and cartoon accessories used in the image, and output them as standard JSON format data. If there is no cartoon image, output a null value. The output example is as follows: {"Cartoon image": [],"Cartoon scene description": [], "Cartoon style": [], "Cartoon background color": [], "Cartoon accessories": []}".

[0045] 3) "Please extract the cartoon image, cartoon scene description, cartoon style, cartoon background color, and cartoon accessories used in the image, and output them as standard JSON format data. If there is no cartoon image, all output null values. The output sample is as follows: {"Cartoon image": [], "Cartoon scene description": [], "Cartoon style": [], "Cartoon background color": [], "Cartoon accessories": []}, set the definition of cartoon: Cartoon is a visual art form characterized by simplification and exaggeration, usually including paintings, illustrations or animations, with bright colors, simple lines and vivid character designs."

[0046] Furthermore, the steps of multimodal data extraction and label generation in S3 include: 1) Video processing: pass the optimized prompt words in step S2 above to the multimodal large model (such as MiniCPM-V2.6) through the interface, and upload a video to be analyzed. The program will parse the key frames of the video and submit the prompt words and key frame image data to the large model. The video content will be extracted according to the output dimension specified in the prompt words, and a cartoon image, cartoon scene description, cartoon style, cartoon background color, and cartoon accessory label will be generated in JSON format. 2) Image processing: The optimized prompt words in step S2 are passed to the multimodal large model (such as MiniCPM-V2.6) through the interface, and a picture file to be analyzed is uploaded. The program submits the prompt words and image data to the large model, extracts the image content according to the output dimension specified in the prompt words, and generates cartoon images, cartoon scene descriptions, cartoon styles, cartoon background colors, and cartoon accessory labels in JSON format; 3) Audio processing: Use speech recognition models (such as Whisper-large) to transcribe audio content and extract speech information from the audio. For music audio, we use music analysis models to extract information such as the pitch, intensity, rhythm, and spectrum of the audio to form structured text content, which is then stored. 4) Text analysis: Calculate the text content based on the large language model, provide the structured text content output in the above step 3) to the text large language model (such as DeepSeek), and extract content labels according to different dimensions based on the given prompt words (another set of prompt words extracted for the text content), and generate structured label data.

[0047] Furthermore, the specific method of label storage and management in step S4 is to obtain the original label data through the model calculation in step S3, parse the original data, obtain the key-value structure in the cognitive, emotional, narrative, atmosphere, and value judgment analysis dimensions (such as JSON structure data in the long text output by the model), and form structured data. The structured data is the required label data, and then the label data is stored in a relational database and indexed by category.

[0048] Through the above method, the content of four modal data, video, image, voice, and text, is extracted. The process is completely based on the given prompt words. Subsequent users can flexibly switch the analysis dimensions and extract different labels without any coding during the analysis process. This solution is based on a large language model, and provides an efficient and intelligent multimodal data processing method for fields such as social sciences and journalism and communication by simplifying the operating process, lowering the technical threshold, and enhancing the flexibility of data analysis. The solution of the present invention can also process images. The image processing method is the same as the video processing method. The video processing process will be converted into an image through frame extraction, and the image processing reduces the frame extraction link.

[0049] Specific usage process: The following takes the multimodal data content extraction method based on prompt words for a single data as an example to illustrate the page operation and the output tag structure. The same is true for batch data analysis. It is a background running program without a page display part.

[0050] 1. Upload the image / video for analysis and extract the result data page based on the prompt words. The prompt words are as follows: Figure 6 , then perform multi-modal debugging such as Figure 7 .

[0051] 2. The label table data page formed after labeling and parsing the original data, the specific interface diagram Figure 8 .

[0052] 3. Submit the speech-to-text page (the example data content is different), and the interface is displayed as follows Fig. 9 .

[0053] 4. Extract labels from the above speech-to-text conversion results. The prompt words are as follows: "Please extract the topics, scenes, disease names, medical terms, symptoms, treatment methods, drug names, and adjectives and nouns that describe a disease in the content, and output them in JSON format. If there are none, output a null value." The output result is as follows. After parsing the result, you can get structured label data such as Fig.10 .

[0054] 0 Views Figure 5 In an optional embodiment, the present invention provides a multimodal data content extraction system based on prompt words, the system comprising: The prompt word design and optimization module can continuously clarify the key information that needs to be extracted and improve the accuracy of large model recognition; Multimodal data extraction and label generation module, which can analyze, extract and generate labels for video, image, audio and text data; The tag storage and management module uses a large language model to calculate multimodal data to obtain the original content description data. We parse the original content description data to obtain the key-value structure in the cognitive, emotional, narrative, atmosphere, and value judgment analysis dimensions (such as the JSON structure data in the long text output by the model), thereby forming structured data. The structured data is the tag data we need, and then the tag data is stored in a relational database (such as MySQL) and indexed by category; The batch data processing module, based on the optimized prompt words in the prompt word design and optimization module, circulates the data to be processed according to the above multimodal data extraction and label generation module and the label storage and management module, and stores multiple label data in a relational database (such as MySQL) in turn according to the results of each group of data output, and indexes them by category.

[0055] Furthermore, the prompt word design and optimization module includes a general prompt word template module, a theme customized prompt word module and a loop debugging and optimization module; The general prompt word template module is used to design general prompt words, which are suitable for basic label extraction of multimodal data (such as action, scene, character, text, etc.); Theme-customized prompt word module optimizes prompt words to analyze the dimensions of cognition, emotion, narrative, atmosphere, and value judgment, and supports data labeling of the above dimensions; The loop debugging and optimization module performs multiple rounds of optimization on the prompt words through the prompt word debugging interface until the analysis requirements are met.

[0056] The working process of the prompt word design and optimization module is explained through a specific case. For example, the following is a prompt word that has been optimized in three steps. It is necessary to extract the cartoon elements that appear in the video. The first step roughly gives the dimensions that need to be extracted, the second step clarifies the content that needs to be extracted, and the third step continues to add descriptions and definitions of the extracted content. After these three steps of optimization, the key information that needs to be extracted is continuously clarified, and the accuracy of recognition through the large model is also continuously improved.

[0057] 1) Please extract the cartoon elements used in the image and output them as standard JSON format data.

[0058] 2) "Please extract the cartoon image, cartoon scene description, cartoon style, cartoon background color, and cartoon accessories used in the image, and output them as standard JSON format data. If there is no cartoon image, output a null value. The output example is as follows: {"Cartoon image": [],"Cartoon scene description": [], "Cartoon style": [], "Cartoon background color": [], "Cartoon accessories": []}".

[0059] 3) "Please extract the cartoon image, cartoon scene description, cartoon style, cartoon background color, and cartoon accessories used in the image, and output them as standard JSON format data. If there is no cartoon image, all output null values. The output sample is as follows: {"Cartoon image": [], "Cartoon scene description": [], "Cartoon style": [], "Cartoon background color": [], "Cartoon accessories": []}, set the definition of cartoon: Cartoon is a visual art form characterized by simplification and exaggeration, usually including paintings, illustrations or animations, with bright colors, simple lines and vivid character designs."

[0060] Furthermore, the multimodal data extraction and label generation module includes a video processing module, an image processing module, an audio processing module and a text analysis module; The video processing module passes the optimized prompt words in the prompt word design and optimization module to the multimodal large model (such as MiniCPM-V2.6) through the interface, and uploads a video to be analyzed. The program will parse the key frames of the video and submit the prompt words and key frame image data to the large model. The video content is extracted according to the output dimension specified in the prompt words, and a cartoon image, cartoon scene description, cartoon style, cartoon background color, and cartoon accessory label are generated in JSON format; The image processing module passes the optimized prompt words in the prompt word design and optimization module to the multimodal big model through the interface, and uploads a picture file to be analyzed. The program submits the prompt words and image data to the big model, extracts the image content according to the output dimension specified in the prompt words, and generates cartoon images, cartoon scene descriptions, cartoon styles, cartoon background colors, and cartoon accessory labels in JSON format; The audio processing module uses a speech recognition model to transcribe the audio content and extract the speech information in the audio. For music audio, we use a music analysis model to extract the audio pitch, intensity, rhythm, spectrum and other information to form structured text content, and store the structured text content. The text analysis module calculates the text content based on the large language model, provides the structured text content output from the above audio processing module to the text large language model (such as DeepSeek), and extracts content tags according to different dimensions based on given prompt words (another set of prompt words extracted for text content), and generates structured tag data.

[0061] In an optional embodiment, the present invention further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement Figure 1-4 The method in .

[0062] In an optional embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement Figure 1-4 The method in .

[0063] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0064] It is understandable that the related features in the above methods and systems can be referenced to each other. In addition, the "first", "second" and the like in the above embodiments are used to distinguish the embodiments, and do not represent the advantages and disadvantages of the embodiments.

[0065] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0066] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the application is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the application described herein, and the description of the specific language above is for the purpose of disclosing the best mode of implementation of the application.

[0067] In addition, the memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0068] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0069] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0070] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0072] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0073] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0074] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0075] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0076] Those skilled in the art should understand that the computing model provided in the embodiments of the present application is not limited to MiniCPM, Whisper, and DeepSeek, and that the effects of the present invention may be achieved by replacing it with other computing models in the same field. The above three models are merely a preferred embodiment of the present invention.

[0077] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent substitution or improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for extracting content from multimodal data based on prompt words, characterized in that: The method comprises: S1, configure the operating environment and provide the basis for extracting the prompt word content; S2, design and optimize prompt words, including designing general prompt word templates, theme customization of prompt words, and loop debugging and optimization of prompt words; S3, multimodal data extraction and label generation, including extracting content labels for videos, images, audio, and text; S4, tag storage and management, uses a large language model to calculate multimodal data to obtain original content description data, parses the original data, stores it in a relational database, and indexes it by category; S5, batch data processing, after completing the calculation of a single prompt word and a single video, image, audio and text, based on the prompt word optimized in step S2, according to the above steps S3 and S4, the data to be processed is looped, and according to the results of each group of data output, multiple label data are stored in the relational database in turn, and indexed by category.

2. The method for extracting content from multimodal data based on prompt words according to claim 1, characterized in that: The steps for configuring the operating environment in S1 include: 1) Prepare a high-performance computer host for deploying a large language model environment; 2) Build a visual recognition environment based on an open source multimodal large model and provide an API interface through coding; 3) Build a speech recognition environment based on the open source speech model and provide an API interface through coding; 4) Build a large model text reasoning environment based on the open source text large language model, and provide an API interface through coding; 5) Develop a program that integrates the API interfaces of the above three steps and provides an interface for user operations.

3. The method for extracting multimodal data content based on prompt words according to claim 1, characterized in that: The steps of designing and optimizing prompt words in S2 include: 1) General prompt word template: design general prompt words suitable for basic label extraction of multimodal data; 2) Theme-customized prompt words: Based on the dimensions of cognition, emotion, narrative, atmosphere, and value judgment analysis, the prompt words are optimized to analyze these dimensions and support data labeling of the above dimensions; 3) Cycle debugging and optimization: perform multiple rounds of optimization on the prompt words through the prompt word debugging interface until the analysis requirements are met.

4. The method for extracting content from multimodal data based on prompt words according to claim 1, characterized in that: The steps of multimodal data extraction and label generation in S3 include: 1) Video processing: The optimized prompt words in step S2 are passed to the multimodal big model through the interface, and a video to be analyzed is uploaded. The program will parse the key frames of the video and submit the prompt words and key frame image data to the big model. The video content is extracted according to the output dimension specified in the prompt words, and structured tags are generated in JSON format. 2) Image processing: The optimized prompt words in step S2 are passed to the multimodal big model through the interface, and a picture file to be analyzed is uploaded. The program submits the prompt words and image data to the big model, extracts the image content according to the output dimension specified in the prompt words, and generates structured tags in JSON format; 3) Audio processing: Use the speech recognition model to transcribe the audio content and extract the speech information in the audio. For music audio, use the music analysis model to extract the audio pitch, intensity, rhythm, and spectrum information to form structured text content, and store the structured text content. 4) Text analysis: Calculate the text content based on the large language model, provide the structured text content output in the above step 3) to the text large language model, and extract content tags according to different dimensions based on the given prompt words, and generate structured tag data.

5. The method for extracting content from multimodal data based on prompt words according to claim 1, characterized in that: The specific method of label storage and management in step S4 is to obtain the original label data through the model calculation in step S3. We parse the original data to obtain the key-value structure in the cognitive, emotional, narrative, atmosphere, and value judgment analysis dimensions to form structured data. The structured data is the required label data. The label data is then stored in a relational database and indexed by category.

6. A multimodal data content extraction system based on prompt words, characterized in that: The system comprises: The prompt word design and optimization module can continuously clarify the key information that needs to be extracted and improve the accuracy of large model recognition; Multimodal data extraction and label generation module, which can analyze, extract and generate labels for video, image, audio and text data; The tag storage and management module uses a large language model to calculate multimodal data to obtain the original content description data. We parse the original content description data to obtain the key-value structure in the cognitive, emotional, narrative, atmosphere, and value judgment analysis dimensions, thereby forming structured data. The structured data is the required tag data, which is then stored in a relational database and indexed by category. The batch data processing module, based on the optimized prompt words in the prompt word design and optimization module, circulates the data to be processed according to the above multimodal data extraction and label generation module and the label storage and management module, and stores multiple label data in the relational database in turn according to the results of each group of data output, and indexes them by category.

7. A system for extracting multimodal data content based on prompt words as claimed in claim 6, characterized in that: The prompt word design and optimization module includes a general prompt word template module, a theme customized prompt word module and a loop debugging and optimization module; The general prompt word template module is used to design general prompt words, which is suitable for basic label extraction of multimodal data; Theme-customized prompt word module optimizes prompt words to analyze the dimensions of cognition, emotion, narrative, atmosphere, and value judgment, and supports data labeling of the above dimensions; The loop debugging and optimization module performs multiple rounds of fine-tuning and optimization on the prompt words through the prompt word debugging interface until the analysis requirements are met.

8. The multimodal data content extraction system based on prompt words as claimed in claim 6, characterized in that: The multimodal data extraction and label generation module includes a video processing module, an image processing module, an audio processing module, and a text analysis module; The video processing module passes the optimized prompt words in the prompt word design and optimization module to the multimodal big model through the interface, and uploads a video to be analyzed. The program will parse the key frames of the video and submit the prompt words and key frame image data to the big model. It will extract the video content according to the output dimension specified in the prompt words and generate structured tags in JSON format. The image processing module passes the optimized prompt words in the prompt word design and optimization module to the multimodal big model through the interface, and uploads a picture file to be analyzed. The program submits the prompt words and image data to the big model, extracts the image content according to the output dimension specified in the prompt words, and generates structured tags in JSON format; The audio processing module uses a speech recognition model to transcribe the audio content and extract the speech information in the audio. For music audio, we use a music analysis model to extract the pitch, intensity, rhythm, and spectrum information of the audio to form structured text content, and store the structured text content; The text analysis module calculates the text content based on the large language model, provides the structured text content output from the above audio processing module to the text large language model, and extracts content tags according to different dimensions based on the given prompt words, and generates structured tag data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images