Multimedia material label management method and system based on multi-modal large model

Abstract

The application provides a multimedia material label management method and system based on a multi-modal large model, which first performs hierarchical structural processing on multimedia material text information to obtain structured text information containing a text content hierarchical structure and semantic association information, then performs label group matching based on a preset tree-shaped label system to determine an associated target label group, then calls a multi-modal large model to execute a labeling scheme according to a target label group labeling scheme attribute to generate a label value labeling result and a weight value, integrates the labeling result and the weight value and performs screening and sorting to generate a final label set, and finally performs versioned maintenance management on the tree-shaped label system and the labeling scheme, thereby improving the efficiency, accuracy and adaptability of multimedia material label management and meeting the requirements of rapid retrieval and accurate query.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a multimedia material label management method and system based on a multi-modal large model. BACKGROUND

[0002] In government agencies, companies and other types of organizations, a large number of multimedia materials are generated and used every day, which include text documents, Word documents (including doc, docx, wps, etc.), PDF documents, PPT documents, Excel spreadsheets, web pages, pictures, videos, audios and other types. In the face of such a large amount of multimedia materials, how to achieve efficient and accurate query and retrieval to meet the diverse needs of organizations and their stakeholders has become a key problem to be solved.

[0003] The traditional manual retrieval method mainly relies on manually browsing the content of the materials one by one to find the required information. This method not only consumes a lot of manpower, material resources and time, resulting in extremely low retrieval efficiency, but also due to the limitations of manual operation, it is easy to miss the detection, and cannot guarantee the integrity and accuracy of the retrieval results. In order to improve the retrieval efficiency, there are currently methods for labeling multimedia materials with keywords, for example, the invention patent CN113486197A uses a multi-user cooperative manual editing method to complete the maintenance of multimedia material labels. However, this manual labeling method also has many drawbacks, the speed of manual operation is limited, it is difficult to meet the demand of rapid labeling of large-scale multimedia materials, resulting in low efficiency; different personnel may have different understanding and labeling standards for the labels, it is difficult to guarantee the consistency and accuracy of the labels; and with the continuous emergence of new materials and the continuous change of business needs, the required labels need to be frequently updated and iterated, and the label management method based on manual cooperation is difficult to respond to these changes in a timely manner, and cannot meet the needs of actual business scenarios. Therefore, there is an urgent need for a more efficient, accurate and flexible multimedia material label management method. SUMMARY

[0004] In view of the above-mentioned problems, in combination with the first aspect of the present application, the embodiments of the present application provide a multimedia material label management method based on a multi-modal large model, which comprises:

[0005] performing hierarchical structured processing on the text information of the multimedia material to obtain structured text information with a tree-like hierarchical relationship, the structured text information containing a text content hierarchical structure and semantic association information of the corresponding hierarchy;

[0006] Based on a preset tree-shaped label system, the structured text information is subjected to label group matching processing, a target label group associated with the structured text information is determined, and the target label group comprises at least one label value and a labeling scheme attribute corresponding to each label value;

[0007] According to the labeling scheme attribute of the target label group, a multi-modal large model is called to execute a corresponding labeling scheme, the structured text information is subjected to label value identification processing, a labeling result of each label value and a corresponding weight value are generated;

[0008] The labeling results and weight values of the target label groups are integrated, the labeling results are subjected to screening and sorting processing according to the attributes of the label groups, and a final label set of the multimedia material is generated;

[0009] The tree-shaped label system and the labeling scheme are subjected to versioned maintenance management, and label system change information, labeling scheme change information and label result version information are recorded.

[0010] In another aspect, the embodiment of the present application also provides a multimedia material label management system based on a multi-modal large model, comprising a processor, a machine readable storage medium, the machine readable storage medium is connected with the processor, the machine readable storage medium is used for storing programs, instructions or codes, and the processor is used for executing the programs, instructions or codes in the machine readable storage medium to realize the above-mentioned method.

[0011] Based on the above aspects, by performing hierarchical structured processing on the text information of the multimedia material, a structured text information with a tree hierarchical relationship is constructed, which not only contains the hierarchical structure of the text content, but also clearly defines the semantic association information between each level. Based on the preset tree label system, the structured text information is processed by label group matching, which can quickly and accurately determine the target label group associated with the structured text information. The target label group contains at least one label value and the labeling scheme attribute corresponding to each label value. According to the labeling scheme attribute of the target label group, the multi-modal large model is called to execute the corresponding labeling scheme, and the structured text information is processed by label value identification to generate the labeling result and the corresponding weight value of each label value. The multi-modal large model can more accurately identify the label value in the text by virtue of its powerful semantic understanding and processing capability, and the reliability of each label value is reflected through the weight value, further improving the accuracy of label identification. The labeling results and weight values of the target label groups are integrated, and the labeling results are sorted and processed according to the attributes of the label groups to generate the final label set of the multimedia material. The information of multiple label groups is considered comprehensively, so that the final label set is more comprehensive, accurate and orderly, which can better meet the needs of quick retrieval and accurate query of multimedia materials. At the same time, the tree label system and the labeling scheme are maintained and managed in a versioned manner, and the label system change information, the labeling scheme change information and the label result version information are recorded, which facilitates the tracing and optimization of the label management system, and can timely adapt to the generation of new materials and the update and iteration of labels, ensuring the continuous effectiveness and flexibility of the label management method, thereby significantly improving the efficiency, accuracy and adaptability of the multimedia material label management. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is the execution flow diagram of the multimedia material label management method based on the multi-modal large model provided by an embodiment of the application.

[0013] Figure 2 is a schematic diagram of exemplary hardware and software components of the multimedia material label management system based on the multi-modal large model provided by an embodiment of the application. DETAILED DESCRIPTION

[0014] The application will be described in detail below with reference to the accompanying drawings, Figure 1 is a flow diagram of the multimedia material label management method based on the multi-modal large model provided by an embodiment of the application, and the following will introduce the multimedia material label management method based on the multi-modal large model in detail.

[0015] Step S110: hierarchically processing the text information of the multimedia material to obtain structured text information with a tree-like hierarchical relationship, the structured text information including a text content hierarchical structure and semantic association information corresponding to the hierarchical structure.

[0016] In actual multimedia material management application scenarios, the types of multimedia materials are diverse, covering documents (such as word documents, pdf documents, ppt documents, excel spreadsheets, web pages), pictures, audio, and video, etc. For different types of multimedia materials, specific methods need to be used to extract the text information therein.

[0017] Step S111: extracting corresponding text information for different types of multimedia materials, the text information including text content, table content, and text information in pictures of document materials, speech transcription text of audio materials, audio transcription text and key frame text information of video materials.

[0018] For document materials, for word documents, pdf documents, ppt documents, excel spreadsheets, and web page materials, professional document format conversion tools such as pandoc and convertio are used to extract text content and table content, and save the pictures therein. Taking a word document containing product introduction as an example, the text content in the document can be accurately extracted by the pandoc tool. For table content, structured text is saved, which can be a special format that can represent and restore hierarchical structure information, such as html, markdown, json, csv, tsv, xml, yaml, latex, etc. For example, the table header and cell content are saved in json format according to a certain hierarchical relationship, which is convenient for subsequent processing.

[0019] Then, for the saved pictures, image recognition technologies such as optical character recognition (OCR) and multi-modal large models are used to extract text information, picture theme information, picture index numbers, and picture content information. For example, a product picture can be recognized by OCR technology to identify the model number and specifications of the product in the picture, and a multi-modal large model can be used to analyze the picture theme as product display and the picture index number in the document, etc. Then, the information extracted from the picture is inserted into the original material context text in the form of structured text, and the entire text after the insertion of the structured text of the table content (if any) and the structured text of the picture information (if any) is taken as the text information of the document material.

[0020] For picture materials, image recognition technology is used to process picture type materials, extract text information, picture theme information, picture content information and possible table information, and save them as structured text as text information. For example, an image recognition technology will first perform grayscale, noise reduction and other preprocessing on a picture containing a medical chart, then recognize the text and data in the chart, analyze the theme of the chart, which is about the incidence rate of a certain disease, and save these information in xml format to form the text information of the picture material.

[0021] For audio materials, speech recognition technology is used to extract text information from audio files. In processing an audio about academic lectures, the speech recognition system will first perform frame segmentation, windowing and other preprocessing operations on the audio, then convert the audio signal into a feature vector, and through the matching of acoustic model and language model, the speech content is transcribed into text. For example, the lecture content about an academic theory is recognized, and these texts are used as the text information of the audio material.

[0022] For video materials, video decoding tools such as ffmpeg are used to decompose video files into audio and key frame sequences. For a product promotion video, ffmpeg will separate the audio stream and video frames in the video. Then the extracted audio is transcribed into text using speech recognition technology, and the key frame sequence is extracted using image recognition technology to extract text information, picture theme information, picture index number and picture content information, and saved as structured text. For example, the function introduction of the product is extracted from the audio, and the parameter information on the product display picture is extracted from the key frame, and the audio transcription text and the structured text extracted from the key frame are combined as the text information of the video material.

[0023] Step S112: converting the extracted unstructured text content into a structured format, which is a markup language format capable of representing hierarchical relationships.

[0024] After extracting the unstructured text content, it needs to be converted into a structured format for subsequent hierarchical division and semantic analysis. Taking an article about the development of science and technology as an example, the article is originally in the form of ordinary text paragraphs, which belongs to unstructured content. In order to convert it into a structured format, html format is selected. According to the structure of the article's paragraphs, titles, etc., the title of the article is represented by h1-h6 tags to represent different levels of titles, and the paragraphs are represented by p tags. For example, the main title of the article is represented by h1 tag, the sub-title is represented by h2 tag, and the specific paragraph content is represented by p tag. In this way, the originally unstructured text is converted into an html document with hierarchical structure, which is convenient for subsequent computer processing and analysis.

[0025] Step S113: performing hierarchical division processing on the converted structured text, determining hierarchical nodes according to the logical relationship of the text content, each hierarchical node containing a node identifier and a corresponding text segment.

[0026] The converted structured text is hierarchically divided according to the logical relationship of the text content. Taking a technical report containing multiple chapters and sections as an example, the chapter title can be used as a first-level node, the section title under each chapter as a second-level node, and the paragraph under the section as a third-level node, etc. Each node is assigned a unique node identifier, for example, using the level and order number of the node, such as chapter node identifier 1.0, first section node identifier 1.1, and second section node identifier 1.2, etc. Each node contains a corresponding text segment, such as the first-level node corresponding to the text content summary of the entire chapter, the second-level node corresponding to the specific content of the section, and the third-level node corresponding to the detailed description of the paragraph.

[0027] Step S114: identifying semantic association information between different hierarchical nodes, including the inclusion relationship between parent nodes and child nodes, the parallel relationship between sibling nodes, and the cross-hierarchical node reference relationship.

[0028] After determining the hierarchical nodes, the semantic association information between different hierarchical nodes is identified. In the above technical report, the chapter title node is the parent node of the section title node, and the section title node is contained under the chapter title node, which reflects the inclusion relationship between parent nodes and child nodes. Different section title nodes under the same chapter have a parallel relationship, which collectively constitutes the content of the chapter. In a paragraph, the content of other chapters is referenced, thus forming a cross-hierarchical node reference relationship. These semantic association information can be determined through semantic analysis of the text content, such as identifying keywords, context relationships, etc. For example, when a statement such as "as described in Chapter 2" appears in a paragraph, it can be determined that there is a reference relationship between the paragraph node and the second chapter node.

[0029] Step S115: integrating the hierarchical nodes and semantic association information into a tree data structure to generate structured text information containing the hierarchical structure of the text content and the semantic association information.

[0030] The hierarchical nodes and semantic association information are integrated into a tree data structure. Taking a technical report as an example, the root node represents the theme of the entire report, each chapter node is a first-level child node unfolded from the root node, the section node is a second-level child node unfolded from the corresponding chapter node, and the paragraph node is a third-level child node unfolded from the corresponding section node. The connection between nodes represents semantic association information, such as a solid line connection for a containing relationship and a dashed line connection for a reference relationship. Through this tree data structure, a structured text information containing the hierarchical structure of the text content and the semantic association information is generated, facilitating subsequent label group matching and tagging processing.

[0031] Step S120: performing label group matching processing on the structured text information based on the preset tree-shaped label system to determine a target label group associated with the structured text information, the target label group containing at least one label value and a tagging scheme attribute corresponding to each label value.

[0032] Step S121: parsing the preset tree-shaped label system to obtain a hierarchical structure of the tree-shaped label system, the hierarchical structure containing a root label group, a child label group and a label value, each label group and label value containing attribute information.

[0033] The preset tree-shaped label system has a clear hierarchical structure, containing a root label group, a child label group and a label value. Taking a medical literature label system as an example, the root label group is “medical field”, under which there are child label groups “disease”, “treatment method”, “target reader” and the like. Each label group has corresponding attribute information, such as the “disease” label group having a “max_n” attribute of 6, indicating that a maximum of 6 label values can be selected under this label group; the “tagging scheme” attribute is “keyword matching”, specifying the tagging manner of the label values under this label group; the “priority” attribute is “medium”, used to determine the importance of the label values under this label group in sorting. Each label group has a different number of label values, and each label value also has its own attribute information. For example, the “small cell carcinoma” label value under the “disease” label group has a “must tag” attribute of no, indicating that it is not necessarily tagged; the “tagging scheme” attribute is “association rule”, different from the tagging scheme of the parent label group; the “priority” attribute is “high”, indicating that it has a high importance in sorting.

[0034] Step S122: extracting hierarchical structure features of the structured text information, the hierarchical structure features including the number of hierarchical nodes, the depth distribution of nodes and the type of semantic association.

[0035] In order to perform label group matching, hierarchical structure features of structured text information need to be extracted. Taking a medical research paper as an example, the number of chapter nodes, section nodes and paragraph nodes is counted as the number of hierarchical nodes, and the depth of different hierarchical nodes is analyzed, such as the depth of chapter nodes is 1, the depth of section nodes is 2, etc. to obtain the node depth distribution. The semantic association types between nodes are identified, such as the inclusion relationship between chapters and sections, the parallel relationship between different sections, and the reference relationship when a paragraph references the content of other chapters. Through the extraction of these hierarchical structure features, the structure and logical relationship of the text can be comprehensively obtained.

[0036] Step S123: Calculate the matching degree of the hierarchical structure features of the structured text information and each label group in the tree-like label system, which is determined based on hierarchical node quantity similarity, node depth distribution similarity and semantic association type coincidence degree.

[0037] In calculating the matching degree, hierarchical node quantity similarity, node depth distribution similarity and semantic association type coincidence degree are considered respectively. For hierarchical node quantity similarity, the number of hierarchical nodes of the structured text information is compared with the number of nodes of the typical text structure represented by each label group in the tree-like label system to calculate the closeness of the two. For example, there are 5 chapter nodes, 15 section nodes and 30 paragraph nodes in the text, and a certain label group corresponds to a typical text structure with 4 chapter nodes, 12 section nodes and 25 paragraph nodes. By calculating the difference and proportional relationship of the number of nodes, the hierarchical node quantity similarity is obtained.

[0038] For node depth distribution similarity, the depth distribution of different hierarchical nodes in the text is analyzed and compared with the expected depth distribution of the label group. For example, most of the section nodes in the text have a depth of 2, while the label group expects a higher proportion of section nodes with a depth of 2. The closer the distribution of the two, the higher the node depth distribution similarity.

[0039] For semantic association type coincidence degree, the coincidence of the semantic association types between nodes in the text and the semantic association types defined by the label group is counted. If the text mainly contains inclusion and parallel relationships, and the label group also defines these two relationships as the main ones, then the semantic association type coincidence degree is high. The similarities of these three aspects are considered comprehensively, and the matching degree of the structured text information and each label group is calculated by weighted summation or other methods.

[0040] Step S124: Select the label group with the highest matching degree as the initial target label group according to the matching degree sorting result, and the initial target label group contains at least one sub-label group or label value.

[0041] According to the calculated matching degree, each label group in the tree label system is sorted, and the label group with the highest matching degree is selected as the initial target label group. In the example of a medical research paper, if the "disease" label group has the highest matching degree with the structured text information of the paper, then the "disease" label group is the initial target label group. This label group may contain sub-label groups, such as "cancer type", "infectious disease type", etc., or label values, such as "small cell carcinoma", "pneumonia", etc.

[0042] Step S125: Extract the tagging scheme attribute of each label value in the initial target label group, which indicates the specific method for identifying the label value, and determine the initial target label group containing the label value and the tagging scheme attribute as the final target label group.

[0043] For each label value in the initial target label group, its tagging scheme attribute is extracted. For example, the "small cell carcinoma" label value under the "disease" label group has a tagging scheme attribute of "association rule", indicating that the "small cell carcinoma" label is to be given to the paper by using a pre-defined association rule; the tagging scheme attribute of the "pneumonia" label value is "keyword matching", indicating that the tagging is to be performed by matching a pre-set keyword. The initial target label group containing the label value and the tagging scheme attribute is determined as the final target label group, so that subsequent label value identification processing of the structured text information can be performed according to the tagging scheme.

[0044] Step S130: According to the tagging scheme attribute of the target label group, call the multi-modal large model to execute the corresponding tagging scheme, and perform label value identification processing on the structured text information to generate the tagging result of each label value and the corresponding weight value.

[0045] Step S131: Call the multi-modal large model to preprocess the structured text information, and convert the structured text information into an input format that can be processed by the model, which includes hierarchical node identification and vectorized representation of the corresponding text segment.

[0046] Before processing the structured text information, the multi-modal large model needs to preprocess it. Taking the structured text information of a medical research paper as an example, first, the hierarchical node identification and the corresponding text segment in the text are extracted. Then, the text segment is converted into a vector representation through word embedding technology. For each text segment of a hierarchical node, each word in it is mapped to a vector, and these word vectors are combined into a text segment vector through methods such as average pooling or maximum pooling. At the same time, the hierarchical node identification is also encoded, and together with the text segment vector, it forms an input format that can be processed by the model. For example, the chapter node identification and section node identification are converted into digital codes, which are spliced with the corresponding text segment vector to form a unified input vector, facilitating the processing of the multi-modal large model.

[0047] Step S132: Select a corresponding labeling scheme according to the labeling scheme attribute of each label value in the target label group, wherein the labeling scheme includes a keyword matching-based labeling scheme, an association rule-based labeling scheme, a semantic understanding-based labeling scheme, and a deep reasoning-based labeling scheme.

[0048] According to the labeling scheme attribute of each label value in the target label group, a suitable labeling scheme is selected. For example, for the "small cell carcinoma" label value under the "disease" label group, the labeling scheme attribute is "association rule", and the association rule-based labeling scheme is selected. For the "pneumonia" label value, the labeling scheme attribute is "keyword matching", and the keyword matching-based labeling scheme is selected.

[0049] Step S133: When the length of the text segment exceeds the input length limit of the multi-modal large model, the hierarchical nodes of the structured text information are split into multiple sub-nodes, each sub-node contains a text segment whose length does not exceed the input length limit, and the labeling scheme is executed for each sub-node.

[0050] When processing a text segment, if its length exceeds the input length limit of the multi-modal large model, splitting processing is required. Taking a long medical review paper as an example, the text segment of a chapter exceeds the input length limit of the model, and the chapter node is split into multiple sub-nodes, each sub-node contains a text segment whose length does not exceed the input length limit. For example, the chapter is split according to the paragraph, and each paragraph is taken as a sub-node. Then the labeling scheme is executed for each sub-node to ensure that the model can normally process the text information.

[0051] Step S134: Execute the corresponding labeling scheme for each label value.

[0052] The keyword matching-based labeling scheme can refer to the following steps:

[0053] Step S1341: Obtain a preset keyword list corresponding to the target label value, wherein the keyword list contains a plurality of feature words related to the label value.

[0054] For the keyword matching-based labeling scheme, first, a preset keyword list corresponding to the target label value is obtained. Taking the "pneumonia" label value as an example, the preset keyword list may contain feature words such as "pneumonia", "pulmonary inflammation", "cough and fever with pulmonary infection", etc. These keywords are closely related to the "pneumonia" label value and are used for matching in the text.

[0055] Step S1342: Traverse the text segment of each hierarchical node in the structured text information, perform word segmentation processing on the text segment, and obtain a plurality of word units.

[0056] The text segments of each hierarchical node in the structured text information are traversed, and each text segment is processed by word segmentation. When processing the text segment of a medical paper, the text segment is split into multiple lexical units using a word segmentation tool. For example, for a text segment describing symptoms "the patient has a cough and fever, and after examination, it is confirmed to be lung inflammation", the word segmentation tool splits the text segment into lexical units such as "patient", "has", "cough", "fever", "symptom", "after", "examination", "confirmed", "is", "lung inflammation".

[0057] Step S1343: The lexical units are accurately matched with the feature words in the preset keyword list, and the positions of the lexical units that are successfully matched in the text segment and the corresponding hierarchical node identifiers are recorded.

[0058] The lexical units obtained by word segmentation are accurately matched with the feature words in the preset keyword list. In the above example, "lung inflammation" is successfully matched with "lung inflammation" in the preset keyword list, and the position of "lung inflammation" in the text segment and the hierarchical node identifier to which the text segment belongs, such as the chapter node identifier and the section node identifier, are recorded.

[0059] Step S1344: The number of times of successful matching of each hierarchical node text segment is counted, and the number of times of successful matching is weighted according to the importance coefficient of the hierarchical node, which is negatively correlated with the depth of the hierarchical node.

[0060] The number of times of successful matching of each hierarchical node text segment is counted. For example, the keyword "pneumonia" is successfully matched 3 times in the text segments of the multiple section nodes under a chapter node. The number of times of successful matching is weighted according to the importance coefficient of the hierarchical node, which is negatively correlated with the depth of the node. For example, the depth of a chapter node is 1, and the importance coefficient is 10; the depth of a section node is 2, and the importance coefficient is 8; the depth of a paragraph node is 3, and the importance coefficient is 6. If the number of times of successful matching of the keyword "pneumonia" under the chapter node is 1, and the number of times of successful matching under the section node is 2, then the weighted number of times of successful matching is 26.

[0061] Step S1345: The keyword matching result of the label value is determined according to the weighted number of times of successful matching, and when the weighted number of times exceeds a preset threshold, it is determined that the keyword matching of the label value is successful.

[0062] The keyword matching result of the label value is determined according to the weighted matching success number. If the preset threshold is 20, the weighted matching success number of the above-mentioned "pneumonia" label value is 26, which exceeds the preset threshold, and it is determined that the keyword matching labeling of the "pneumonia" label value is successful. When the labeling is successful, the preset weight value of the labeling scheme is accumulated into the weight value of the label value. For example, the preset weight value of the "keyword matching" labeling scheme is 5, and the weight value of the "pneumonia" label value is originally 0, and after the labeling is successful, the weight value becomes 5.

[0063] The labeling scheme based on the association rule can refer to the following steps:

[0064] Step S1346: Analyzing the preset association rule corresponding to the target label value, the association rule including a plurality of condition items and the logical relationship between the condition items, and the logical relationship including an and relationship, an or relationship, and a not relationship.

[0065] For the labeling scheme based on the association rule, the preset association rule corresponding to the target label value is analyzed. Taking the "small cell carcinoma" label value as an example, the association rule can be set as: when the two condition items "appearance of specific cell morphological feature description" and "detection index reaching a certain range" are satisfied at the same time, it is determined that the "small cell carcinoma" label can be labeled, and the logical relationship here is an and relationship. Or the association rule is set as: when "small cell carcinoma related typical symptom description" or "specific gene detection result" appears, the label can be labeled, which is an or relationship. If the association rule is to label when "diagnostic basis for excluding small cell carcinoma" does not appear, it is a not relationship.

[0066] Step S1347: Extracting the text segment related to the condition item in the structured text information, and performing semantic analysis on the text segment to determine whether the text segment meets the requirements of a single condition item.

[0067] After the condition items of the association rule are determined, the text segments related to the condition items are extracted from the structured text information. For example, in the association rule of the above-mentioned "small cell carcinoma", if the condition item is "appearance of specific cell morphological feature description", the text segments about cell morphological description are searched in the structured text information of the medical research paper. Then the semantic analysis is performed on these text segments, and the natural language processing technology such as part-of-speech tagging, named entity recognition, etc. is used to analyze whether the text segment accurately describes the specific cell morphological features. For example, it is determined whether the text mentions the cell size, shape, staining characteristics and other information specific to small cell carcinoma to determine whether the text segment meets the requirements of this condition item.

[0068] Step S1348: Combining the satisfaction results of the multiple condition items according to the logical relationship. When the logical relationship is AND, all condition items are satisfied, and the association rule is established. When the logical relationship is OR, at least one condition item is satisfied, and the association rule is established. When the logical relationship is NOT, the specified condition item is not satisfied, and the association rule is established.

[0069] The satisfaction results of the condition items are combined according to the logical relationship between the condition items in the association rule. If the association rule is an AND relationship, such as the "small cell carcinoma" association rule requiring "appearance of specific cell morphological feature description" and "detection index reaching a certain range" at the same time, the association rule is established only when the text segments corresponding to the two condition items are both determined to satisfy the conditions. If it is an OR relationship, such as the association rule being "appearance of typical symptom description related to small cell carcinoma" or "specific gene detection result", the association rule is established as long as one of the condition items corresponding to the text segment satisfies the condition. If it is a NOT relationship, such as "no appearance of diagnostic basis excluding small cell carcinoma", the association rule is established when there is no relevant diagnostic basis excluding small cell carcinoma in the text.

[0070] Step S1349: Recording the judgment process and the text segment position of the association rule, and generating an association rule verification report.

[0071] In the process of judging the association rule, the judgment process and the text segment position are recorded in detail. For example, in the judgment of the "small cell carcinoma" association rule, the process of semantic analysis of each condition item is recorded, as well as the reasons for satisfying or not satisfying the conditions. At the same time, the specific position of the text segment in the structured text information is recorded, such as the chapter, section and paragraph numbers. These information is sorted to generate an association rule verification report, which includes the specific content of the association rule, the judgment result of each condition item, the text segment position, and the conclusion of whether the association rule is established.

[0072] Step S13410: When the association rule is established, it is determined that the association rule labeling of the label value is successful.

[0073] According to the judgment result of the association rule, when the association rule is established, it is determined that the association rule labeling of the label value is successful. For example, the "small cell carcinoma" association rule is established, and it is determined that the "small cell carcinoma" label value labeling is successful. When the labeling is successful, the preset weight value of the labeling scheme is added to the weight value of the label value. For example, the preset weight value of the "association rule" labeling scheme is 8, and the weight value of the "small cell carcinoma" label value is originally 0, and after the labeling is successful, the weight value becomes 8.

[0074] The labeling scheme based on semantic understanding can refer to the following steps:

[0075] Step S13411: input the text segment of structured text information into the multimodal large model, and generate a semantic summary of the text segment through the text understanding module of the model, wherein the semantic summary contains the core theme and key information of the text segment.

[0076] For the tagging scheme based on semantic understanding, the text segment of structured text information is input into the text understanding module of the multimodal large model. Taking a text segment of a medical review paper as an example, the multimodal large model will perform deep semantic analysis on the text segment. First, the text is analyzed for words, phrases, and their parts of speech; then the syntax of the sentence is analyzed to construct the syntax structure of the sentence; finally, the semantic analysis is performed to understand the meaning of the text. Through these analysis processes, the model generates a semantic summary of the text segment, which contains the core theme of the text segment, such as the research progress of a certain disease, and the key information, such as the new treatment method of the disease, the new discovery of the pathogenesis, etc.

[0077] Step S13412: obtain the description text of the target label value, which contains the definition, applicable scenario and feature description of the label value.

[0078] The description text of the target label value is obtained. For example, for the "literature interpretation" label value, its description text may include "in-depth analysis and interpretation of academic literature, extracting core ideas, research methods and conclusions, etc. from the literature", which clearly defines the definition of the label value, the applicable scenario is for academic literature, and the feature description is to perform in-depth analysis and extract core information.

[0079] Step S13413: input the semantic summary and the label value description text into the semantic matching module of the multimodal large model, calculate the semantic similarity between the two, and the semantic similarity is determined based on the cosine distance of word vectors and the semantic role overlap.

[0080] The generated semantic summary and the description text of the target label value are input into the semantic matching module of the multimodal large model. In this semantic matching module, first, the words in the semantic summary and the description text are converted into word vectors. Then the cosine distance between the word vectors is calculated, the closer the cosine distance is to 1, the more similar the two word vectors are, which means that the semantic summary and the description text have a higher similarity in the lexical level. At the same time, the semantic roles in the semantic summary and the description text, such as subject, predicate, object, etc., are analyzed, and the overlap of the semantic roles is calculated. For example, the semantic summary describes "the research explores the treatment method of a certain disease", and the label value description text is "interpretation of the treatment method in academic literature", both of which have the same subject "treatment method" and similar predicate. By counting the overlap of such semantic roles, the semantic role overlap is obtained. By combining the cosine distance of the word vectors and the semantic role overlap, the semantic similarity between the semantic summary and the label value description text is determined.

[0081] Step S13414: When the semantic similarity exceeds the preset threshold, further analyze whether the semantic summary contains the key features in the label value description text through the multi-modal large model.

[0082] When the calculated semantic similarity exceeds the preset threshold, further analyze whether the semantic summary contains the key features in the label value description text through the multi-modal large model. For example, in the example of the "literature interpretation" label value, the key features of the label value description text are "in-depth analysis" and "extracting core information", and the multi-modal large model will search the semantic summary for whether there are sentences reflecting in-depth analysis, such as detailed analysis of research methods, and whether core information is extracted, such as summarizing the main conclusions of the research.

[0083] Step S13415: If the key features are contained, it is determined that the semantic understanding of the label value is successfully labeled, and the semantic similarity value is recorded as the basis for calculating the weight value.

[0084] If the semantic summary contains the key features in the label value description text, it is determined that the semantic understanding of the label value is successfully labeled. For example, in the above example of "literature interpretation", if the semantic summary reflects both in-depth analysis and extracts core information, it is determined that the "literature interpretation" label value is successfully labeled. When the labeling is successful, the semantic similarity value is recorded as the basis for calculating the weight value. For example, the semantic similarity value is 0.8, and the "semantic understanding" labeling scheme will calculate the weight value according to the similarity value according to the set rules and add it to the weight value of the label value.

[0085] The labeling scheme based on deep reasoning can refer to the following steps:

[0086] Step S13416: Pre-set the deep thinking process and / or logical judgment condition steps required for labeling of a specific label value, and encapsulate the structured text information and the deep thinking process and / or logical judgment condition in the prompt words of the large language model.

[0087] For the labeling scheme based on deep reasoning, the deep thinking process and / or logical judgment condition steps required for labeling of a specific label value are pre-set. Taking the "monthly sales increase" label value as an example, the judgment condition is "the monthly sales of the same product increase month by month for a certain number of consecutive months". The structured text information and these deep thinking processes and logical judgment conditions are encapsulated in the prompt words of the large language model. For example, the prompt word can be expressed as "Please judge whether there is a situation of the monthly sales of the same product increasing month by month for a certain number of consecutive months according to the following text information, the text information is ……", and the structured text information containing product sales data is filled in the ellipsis.

[0088] Step S13417: The large language model is required to analyze and judge each step based on the structured text information, according to the deep thinking process and / or logical judgment condition step, and give the factual basis in the structured text information and the thinking reasoning process of each step.

[0089] After receiving the prompt word, the large language model analyzes and judges each step based on the structured text information, according to the set deep thinking process and logical judgment condition step. In the example of "monthly sales increase", the large language model will first search for whether the structured text information contains sales information of the same product for at least a certain number of consecutive months. If relevant information is found, it will further compare the sales of the adjacent two months to determine whether the sales of the latter month are greater than that of the former month. In this process, the large language model will give the factual basis in the structured text information of each step, such as pointing out which text fragments provide sales data, and the thinking reasoning process, such as explaining how to compare and judge based on sales data.

[0090] Step S13418: According to the judgment result of the large language model, determine whether to mark the specific label value, when the judgment result is yes, determine that the deep reasoning of the label value is successfully marked, and according to the preset weight value of the marking scheme, accumulate to the weight value of the label value.

[0091] According to the judgment result of the large language model, determine whether to mark the specific label value. In the example of "monthly sales increase", if the large language model judges that there is a case of monthly sales of the same product reaching a certain number of consecutive months, then it is determined that the "monthly sales increase" label value is successfully marked. When the marking is successful, according to the preset weight value of the "deep reasoning" marking scheme, accumulate to the weight value of the label value. For example, the preset weight value of the "deep reasoning" marking scheme is 10, and the weight value of the "monthly sales increase" label value is originally 0, and after successful marking, its weight value becomes 10.

[0092] Step S135: Determine the marking result of the label value according to the execution result of the marking scheme, when the marking is successful, according to the preset weight value of the marking scheme, accumulate to the weight value of the label value, generate a label value recognition result containing the marking result and the weight value.

[0093] After the execution of the corresponding tagging scheme for each label value, the tagging result of the label value is determined according to the execution result of the tagging scheme. If a label value is successfully tagged under one or more tagging schemes, the label value is marked as successfully tagged. For example, the "pneumonia" label value is successfully tagged under the keyword matching tagging scheme, the "small cell carcinoma" label value is successfully tagged under the association rule tagging scheme, the "literature interpretation" label value is successfully tagged under the semantic understanding tagging scheme, and the "monthly sales increase" label value is successfully tagged under the deep reasoning tagging scheme. When successfully tagged, the preset weight value of each tagging scheme is accumulated into the weight value of the label value. For example, the preset weight value of the "pneumonia" keyword matching tagging scheme is 5, the preset weight value of the "small cell carcinoma" association rule tagging scheme is 8, the weight value calculated according to the semantic similarity of the "literature interpretation" semantic understanding tagging scheme is 7, and the preset weight value of the "monthly sales increase" deep reasoning tagging scheme is 10. Finally, the label value identification result containing the tagging result (success or failure) and the weight value is generated, which facilitates subsequent screening and sorting processing.

[0094] Step S140: Integrate the tagging results and weight values of each target label group, and perform screening and sorting processing on the tagging results according to the attributes of the label groups to generate the final label set of the multimedia material.

[0095] Step S141: Collect the tagging results and corresponding weight values of all label values in each target label group, and establish a temporary label value set, which contains label value identification, tagging result status and weight value.

[0096] After completing the tagging processing of the label values in each target label group, the tagging results and corresponding weight values of all label values are collected. Taking the label management of a medical research paper as an example, the tagging results (success or failure) and weight values of all label values under the target label groups such as "disease", "treatment method", "target reader" and the like are collected. A temporary label value set is established, each element in the set containing label value identification such as "pneumonia", "small cell carcinoma", "literature interpretation", "monthly sales increase", tagging result status marked as "success" or "failure", and weight value, i.e. the weight value accumulated after various tagging schemes.

[0097] Step S142: Analyze the attribute information of the target label group, which includes the must-tag attribute, the tolerance number attribute and the priority attribute. The must-tag attribute indicates that at least one label value is required under the label group, the tolerance number attribute indicates the maximum number of label values that can be included under the label group, and the priority attribute indicates the sorting priority of the label value.

[0098] The attribute information of the target label group is parsed. Taking the "disease" label group as an example, the "must hit" attribute is "no", indicating that at least one label value is not required under the label group; the "tolerance number attribute" is 6, indicating that at most 6 label values can be included under the label group; and the "priority attribute" is "medium", indicating that the importance of the label value in the sorting is at a medium level. Each label group has similar attribute information, which is used for subsequent filtering and sorting of the tagging result.

[0099] Step S143: For the label group containing the must hit attribute, if the number of label values with successful tagging results in the temporary set is zero, mark the label group as an abnormal state and record the abnormal information.

[0100] For the label group containing the "must hit" attribute, check the number of label values with successful tagging results in the temporary set. If the "must hit" attribute of a certain label group is "yes", and the number of label values with successful tagging results under the label group in the temporary set is zero, mark the label group as an abnormal state. For example, the "must hit" attribute of the "target reader" label group is "yes", but all label values under the "target reader" label group in the temporary set have failed tagging results, at which time the "target reader" label group is marked as an abnormal state, and the abnormal information, such as the time of abnormal occurrence, the label group name, etc., is recorded, so as to be subsequently investigated and processed.

[0101] Step S144: For the label group containing the tolerance number attribute, the number of label values with successful tagging results in the temporary set is counted, and if the number exceeds the tolerance number attribute value, the label values are sorted according to the priority attribute, the label values with high priority are placed in front, the label values with the same priority are sorted according to the weight value from large to small, and the tolerance number of label values in the front are selected and retained in the temporary set.

[0102] Step S1441: The tolerance number attribute value and the priority attribute value of each label value are read from the attribute information of the target label group, and the priority attribute value is used to indicate the relative importance of the label value.

[0103] For the label group containing the tolerance number attribute, the tolerance number attribute value and the priority attribute value of each label value are read from the attribute information of the label group. For example, the tolerance number attribute value of the "disease" label group is 6, the priority attribute value of the "small cell carcinoma" label value under it is "high", the priority attribute value of the "pneumonia" label value is "medium", etc. The priority attribute value is used to indicate the relative importance of the label value, and plays an important role in sorting.

[0104] Step S1442: The label values with successful tagging results in the temporary set are traversed, the priority attribute value and the weight value of each label value are extracted, and a sorting feature list is established.

[0105] The priority attribute value and the weight value of each label value are extracted from the label values with successful tagging in the temporary set, and a sorted feature list is established. For example, in the "disease" label group, the priority attribute value and the weight value of the label values with successful tagging such as "small cell carcinoma" and "pneumonia" are extracted to form a list, and each element in the list contains a label value identifier, a priority attribute value and a weight value.

[0106] Step S1443: The label values in the sorted feature list are sorted in the first level according to the priority attribute value from high to low, and the label values with the same priority attribute value are grouped into the same priority group.

[0107] The label values in the sorted feature list are sorted in the first level according to the priority attribute value from high to low, and the label values with the same priority attribute value are grouped into the same priority group. For example, in the sorted feature list of the "disease" label group, the priority of "small cell carcinoma" is "high", the priority of "pneumonia" is "medium", and the priority of "liver cancer" is "medium", so "small cell carcinoma" is placed in the front, and "pneumonia" and "liver cancer" are grouped into the same priority group.

[0108] Step S1444: The label values in each priority group are sorted in the second level according to the weight value from large to small, and a group sorting result is generated.

[0109] The label values in each priority group are sorted in the second level according to the weight value from large to small. In the above example of the "disease" label group, the label values in the priority group of "pneumonia" and "liver cancer" are sorted according to the weight value from large to small. Assuming that the weight value of "pneumonia" is 12 and the weight value of "liver cancer" is 9, "pneumonia" is placed in front of "liver cancer", and the sorting result in the priority group is generated.

[0110] Step S1445: The priority groups after the first level sorting are arranged in order, and the label values in each priority group are arranged according to the group sorting result, to form a complete label value sorting sequence.

[0111] The priority groups after the first level sorting are arranged in order, and the label values in each priority group are arranged according to the group sorting result, to form a complete label value sorting sequence. In the "disease" label group, "small cell carcinoma" with the priority of "high" is arranged first, and then "pneumonia" and "liver cancer" with the priority of "medium" and sorted according to the weight value are arranged, to form a complete sorting sequence.

[0112] Step S1446: The first tolerance number of label values are selected from the label value sorting sequence in turn, and are retained in the temporary set, and the remaining label values are removed from the temporary set.

[0113] The first tolerance number of label values from the complete label value sorting sequence are selected in turn and reserved in a temporary set. For example, the tolerance number attribute value of the "disease" label group is 6, the first 6 label values from the sorting sequence are selected and reserved in the temporary set, and the remaining label values are removed from the temporary set. In this way, the screening and sorting of label values under the label group containing the tolerance number attribute are completed.

[0114] Step S145: The label values in the processed temporary set of each target label group are integrated, duplicate label values are removed, and a final label set of the multimedia material is generated, which contains label value identifiers, the corresponding label groups, and weight values.

[0115] After the screening and sorting of label values under each target label group are completed, the label values in the processed temporary set of each target label group are integrated. Taking the label management of a medical research paper as an example, the label values in the processed temporary set of the target label groups "disease", "treatment method", and "target reader" are combined into a new set. In the integration process, duplicate label values need to be removed. For example, the label value "medical research" may appear in different label groups, and only one is retained. The final label set of the multimedia material contains label value identifiers such as "pneumonia", "small cell carcinoma", "literature interpretation", and "drug treatment", the corresponding label groups, which clearly indicate that each label value comes from which label group, such as "pneumonia" belonging to the "disease" label group, and weight values, which are determined after various labeling schemes are accumulated, screened, and sorted. The final label set provides clear and accurate label annotations for the multimedia material.

[0116] Step S150: The tree-shaped label system and the labeling scheme are versioned and maintained, and label system change information, labeling scheme change information, and label result version information are recorded.

[0117] Step S151: Receive a change request for the tree-shaped label system, which contains operation instructions for adding a label group, deleting a label group, modifying a label group attribute, adding a label value, or modifying a label value attribute.

[0118] In the actual multimedia material label management process, the tree-shaped label system may need to be changed according to business needs or knowledge updates, so it can receive change requests for the tree-shaped label system, which contain various operation instructions. For example, there may be a request to add a new label group, such as adding a "new treatment" label group in a medical literature label system; a request to delete a label group when it is no longer applicable, such as the "obsolete treatment method" label group; a request to modify the attributes of a label group, such as changing the "max_n" attribute of the "disease" label group from one value to another; a request to add a label value, such as adding a "rare disease type" label value under the "disease" label group; a request to modify the attributes of a label value, such as changing the "priority" attribute of the "small cell carcinoma" label value from "high" to "medium".

[0119] Step S152: approval processing of the change request, verifying whether the change operation conforms to the preset change rules, and after approval, performing the change operation and generating a new label system version, recording the version identifier, change time, change content, and change executor information, the change rules including label group naming specifications, hierarchical structure constraints, and attribute value range restrictions.

[0120] After receiving the change request for the tree-shaped label system, it needs to be approved, which can verify whether the change operation conforms to the preset change rules. For example, the label group naming specification requires that the label group name cannot contain special characters and has certain semantic relevance; the hierarchical structure constraint specifies that the hierarchical relationship of the label group cannot be arbitrarily disrupted, such as a child label group must belong to a suitable parent label group; the attribute value range restriction specifies that the "max_n" attribute value must be a positive integer, etc. During the approval process, the change request will be thoroughly checked. If the change request conforms to all the change rules, it is approved. After approval, the change operation is performed, such as adding a label group, adding a new node and related attributes in the tree-shaped label system; deleting a label group, removing the corresponding node and its child nodes. At the same time, a new label system version is generated, recording the version identifier, such as using version numbers, which are incremented each time a change is made; recording the change time, accurate to the specific date and time; recording the change content, detailing which change operations were made; recording the change executor information, clearly indicating which user initiated and executed the change operation.

[0121] Step S153: receiving a change request for the labeling scheme, the change request containing operation instructions to add a labeling scheme, modify the logic of the labeling scheme, or adjust the weight value of the labeling scheme.

[0122] In addition to the change of the tree-shaped label system, the marking scheme may also need to be adjusted, and the change request of the marking scheme can be received, which contains various operation instructions. For example, the request of adding a marking scheme, such as introducing a marking scheme based on a new machine learning algorithm; the request of modifying the logic of the marking scheme, such as updating the keyword list in the marking scheme based on keyword matching; the request of adjusting the weight value of the marking scheme, such as adjusting the weight value of the "keyword matching" marking scheme from one value to another value.

[0123] Step S154: Validity check is performed on the change request of the marking scheme, whether the modified marking scheme logic has syntax error or logical conflict is verified, after the check is passed, the change operation is performed and a new marking scheme version is generated, and the version identification, change time, change content and change executor information are recorded.

[0124] For the change request of the marking scheme, validity check is needed, which can verify whether the modified marking scheme logic has syntax error or logical conflict. For example, when modifying the marking scheme based on association rules, check whether the logical operators in the rules are used correctly, and whether the relationship between the condition items is reasonable. If the modified marking scheme logic has syntax error, such as format error of regular expression; or has logical conflict, such as nested use of "and" relationship and "or" relationship leads to incorrect execution of rules, the check is not passed. Only after the check is passed, the change operation is performed, such as adding a new marking scheme to the system and configuring related parameters; updating the corresponding rule code when modifying the logic of the marking scheme; modifying the weight parameter of the marking scheme in the system when adjusting the weight value of the marking scheme. At the same time, a new marking scheme version is generated, and the version identification, change time, change content and change executor information are recorded, which is similar to the version record of the tree-shaped label system, facilitating the tracing and management of the change of the marking scheme.

[0125] Step S155: When the tree-shaped label system or the marking scheme is updated, the affected multimedia materials are identified, the label value identification and integration processing of the affected materials are performed again, a new label result version is generated, and the label result version identification, associated label system version, marking scheme version and generation time are recorded.

[0126] When the tree-shaped label system or the tagging scheme is updated, it is necessary to identify the affected multimedia materials. For example, when a new label value is added under the "disease" label group, all multimedia materials containing disease-related content may be affected; when the logic of the "keyword matching" tagging scheme is modified, all multimedia materials using the tagging scheme for label value identification will be affected. For the affected materials, re-execute the label value identification and integration processing. According to the steps S110-S140 described above, the text information of the affected multimedia materials is hierarchically structured, the label group matching is performed based on the new tree-shaped label system, the label value identification is performed according to the new tagging scheme, and finally the tagging result and weight value are integrated. A new label result version is generated, the label result version identifier is recorded, the corresponding label system version and tagging scheme version are associated, it is clear that the label result is generated under which label system and tagging scheme version, and the generation time is recorded, which facilitates the management and tracing of the historical versions of the label result. Through this versioned maintenance and management, it is ensured that the changes of the tree-shaped label system, the tagging scheme and the label result can be effectively recorded and controlled, and the accuracy and traceability of the multimedia material label management are guaranteed.

[0127] In the entire multimedia material label management process, if private sensitive data such as patient personal information and medical records are involved, a series of privacy protection and anti-leakage technical means will be used. First, in the data collection stage, the data will be encrypted, using symmetric encryption or asymmetric encryption algorithms, to convert the data into ciphertext form for storage and transmission, preventing data from being stolen during transmission. Second, in terms of data storage, access control technology will be used to set different access permissions, only authorized personnel can access and process private sensitive data. At the same time, regular backups of stored data will be made to prevent data loss. During data processing, private sensitive data will be desensitized, such as anonymizing patient names and partially hiding identity card numbers and other key information. For the training of artificial intelligence models, if private sensitive data is used, federated learning and other technologies will be used to train the model without sharing the original data, ensuring data privacy. Through these technical means, private sensitive data is effectively protected to prevent data leakage and misuse.

[0128] In the present multimedia material label management method, the construction and training process of the multi-modal large model is as follows.

[0129] The multi-modal large model mainly includes an input layer, a text understanding module, a semantic matching module, and an output layer. The input layer is responsible for receiving structured text information and label value description texts and converting them into a format that the model can process, such as performing word embedding processing on the texts and encoding the hierarchical node identifiers. The text understanding module is the core part of the model and includes multiple neural network layers, such as convolutional neural networks (CNN), recurrent neural networks (RNN), or their variants (such as LSTM, GRU), etc. These neural network layers perform deep semantic analysis on the input texts, including lexical analysis, syntactic analysis, and semantic understanding, to generate semantic representations of the texts. The semantic matching module receives the semantic representations output by the text understanding module and calculates the semantic similarity between the semantic summaries of the text segments and the label value description texts. It can use a neural network structure based on an attention mechanism to perform weighted calculation on the semantic representations to improve the accuracy of the matching. The output layer outputs the labeling results of the label values based on the semantic similarity and a pre-set threshold. The modules are connected through fully connected layers or skip connections, etc., to ensure effective transmission and processing of information.

[0130] During the training of the multi-modal large model, a training data set needs to be prepared first. The training data set contains a large amount of multimedia material text information and corresponding label value annotation data. The data set is divided into a training set, a validation set, and a test set, and the proportion can be adjusted according to the actual situation. During the training process, stochastic gradient descent (SGD) or its optimization algorithms (such as Adagrad, Adadelta, Adam, etc.) are used for parameter updating. Learning rate, batch size, and training rounds, etc. are set as hyperparameters. The learning rate controls the step size of parameter updating, the batch size determines the number of data samples used in each training, and the training rounds represent the number of times the entire data set is trained. In each training round, the training data is input into the model, and the loss function, such as the cross-entropy loss function, between the model's output and the true label value is calculated. According to the value of the loss function, the gradient is calculated through the backpropagation algorithm, and the parameters of the model are updated. The performance of the model is evaluated regularly on the validation set, and the hyperparameters are adjusted according to the validation results to prevent overfitting or underfitting of the model. Finally, the trained model is evaluated on the test set to ensure that the model has good generalization ability on unknown data.

[0131] In the specific field and scene of multimedia material label management, the multi-modal large model and related algorithms are closely combined with actual application. The input data of the model is set as structured text information and label value description text. The structured text information contains hierarchical node identification and vectorized representation of corresponding text segments, which reflects the text content and hierarchical structure of the multimedia material. The label value description text contains information such as definition, applicable scene and feature description of the label value. The output data of the model is the labeling result and weight value of the label value, which clearly indicates whether each label value can be labeled and its importance. Through the setting of such input and output data, the internal correlation between the model and the multimedia material label management scene is established. For example, in the medical literature label management scene, the model determines whether the corresponding label can be labeled for the literature according to the structured text information of the medical literature and the description text of the disease, treatment method and other label values, and the output labeling result and weight value provide the basis for the classification and retrieval of the literature. At the same time, different labeling schemes such as keyword matching, association rules, semantic understanding and deep reasoning are selected and applied according to the specific business requirements and characteristics of the label value, further enhancing the applicability and accuracy of the model in specific scenes.

[0132] Figure 2 A schematic diagram of exemplary hardware and software components of a multi-modal large model-based multimedia material label management system 100 provided by some embodiments of the present application is shown. For example, a processor 120 can be used in the multi-modal large model-based multimedia material label management system 100 and used to perform the functions in the present application.

[0133] The multi-modal large model-based multimedia material label management system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the multi-modal large model-based multimedia material label management method of the present application. Although only one server is shown in the present application, for the sake of convenience, the functions described in the present application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0134] For example, the multi-modal large model-based multimedia material label management system 100 can include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and different forms of storage media 140, such as a disk, a ROM, or a RAM, or any combination thereof. Exemplarily, the multi-modal large model-based multimedia material label management system 100 can also include program instructions stored in the ROM, the RAM, or other types of non-transitory storage media, or any combination thereof. The methods of the present application can be implemented according to these program instructions. The multi-modal large model-based multimedia material label management system 100 also includes an I / O interface 150 between the computer and other input / output devices.

[0135] For the convenience of description, only one processor is described in the multi-modal large model-based multimedia material label management system 100. However, it should be noted that the multi-modal large model-based multimedia material label management system 100 in the present application can also include multiple processors, so the steps performed by one processor described in the present application can also be jointly performed or separately performed by multiple processors. For example, if the processor of the multi-modal large model-based multimedia material label management system 100 performs steps A and B, it should be understood that steps A and B can also be jointly performed by two different processors or separately performed in one processor. For example, a first processor performs step A, a second processor performs step B, or a first processor and a second processor jointly perform steps A and B.

[0136] In addition, the embodiment of the present application also provides a readable storage medium, wherein computer executable instructions are pre-stored, and when the processor executes the computer executable instructions, the multi-modal large model-based multimedia material label management method is realized.

[0137] It should be noted that, in order to simplify the description of the present application and to help understand one or more embodiments of the present application, in the foregoing description of the embodiments of the present application, various features are sometimes combined into one embodiment, figure or description thereof.

Claims

1. A multi-modal large model-based multimedia material label management method, characterized in that, The method comprises: hierarchical structuring of text information of multimedia material to obtain structured text information with tree hierarchical relationship, the structured text information containing text content hierarchical structure and semantic association information corresponding to the hierarchical structure; based on a preset tree label system, the structured text information is matched with a target label group associated with the structured text information, the target label group containing at least one label value and the corresponding labeling scheme attribute of each label value; According to the labeling scheme attribute of the target label group, the multi-modal large model is called to execute the corresponding labeling scheme, and the label value identification processing is performed on the structured text information to generate the labeling result and the corresponding weight value of each label value; Integrate the labeling results and weight values of each target label group, and sort and process the labeling results according to the attributes of the label groups to generate the final label set of the multimedia material; The tree label system and the labeling scheme are versioned and maintained, and the label system change information, the labeling scheme change information and the label result version information are recorded; The method comprises: Analyzing the preset tree label system to obtain the hierarchical structure of the tree label system, the hierarchical structure containing root label groups, sub-label groups and label values, each label group and label value containing attribute information; Extract the hierarchical structure features of the structured text information, including the number of hierarchical nodes, node depth distribution and semantic association type; Calculate the matching degree of the hierarchical structure features of the structured text information and each label group in the tree label system, the matching degree being determined based on the hierarchical node quantity similarity, node depth distribution similarity and semantic association type coincidence degree; According to the matching degree sorting result, select the label group with the highest matching degree as the initial target label group, the initial target label group containing at least one sub-label group or label value; Extract the labeling scheme attribute of each label value in the initial target label group, the labeling scheme attribute being used to indicate the specific method for identifying the label value, and the initial target label group containing the label value and the labeling scheme attribute is determined as the final target label group. 2.The multi-modal large model-based management method of multimedia material labels according to claim 1, wherein, The method comprises: For different types of multimedia material, extract the corresponding text information, including the text content, table content and picture text information of document material, the speech transcription text of audio material, and the audio transcription text and key frame text information of video material; Convert the extracted unstructured text content into a structured format, which is a markup language format that can represent hierarchical relationships; Hierarchical division processing is performed on the converted structured text, and the hierarchical nodes are determined according to the logical relationship of the text content, each hierarchical node containing a node identifier and the corresponding text segment; The semantic association information between different hierarchical nodes includes the inclusion relationship between parent nodes and child nodes, the parallel relationship between sibling nodes, and the reference relationship between cross-hierarchical nodes. The hierarchical nodes and semantic association information are integrated into a tree data structure to generate structured text information containing the hierarchical structure of the text content and the semantic association information. 3.The multi-modal large model based management method of multimedia material label according to claim 1, wherein, According to the tag value of the target label group, the multi-modal large model is called to execute the corresponding labeling scheme to perform label value identification processing on the structured text information, and generate the labeling result of each label value and the corresponding weight value, including: The multi-modal large model is called to preprocess the structured text information, and the structured text information is converted into an input format that can be processed by the model, which includes hierarchical node identification and vectorization representation of the corresponding text segment; According to the labeling scheme attribute of each label value in the target label group, the corresponding labeling scheme is selected, including keyword matching-based labeling scheme, association rule-based labeling scheme, semantic understanding-based labeling scheme, and deep reasoning-based labeling scheme; When the length of the text segment exceeds the input length limit of the multi-modal large model, the hierarchical nodes of the structured text information are split into multiple sub-nodes, each containing a text segment whose length does not exceed the input length limit, and each sub-node is executed separately; For each label value, the corresponding labeling scheme is executed, and when the labeling scheme is based on keyword matching, the text segment is traversed to match the preset keyword list; When the labeling scheme is the association rule-based labeling scheme, it is verified whether the text segment satisfies the preset logical condition; When the labeling scheme is the semantic understanding-based labeling scheme, the semantic summary of the text segment is generated by the multi-modal large model and matched with the label value description; When the labeling scheme is the deep reasoning-based labeling scheme, the multi-modal large model performs logical analysis on the text segment and determines whether it meets the label value condition; According to the execution result of the labeling scheme, the labeling result of the label value is determined, and when the labeling is successful, the preset weight value of the labeling scheme is accumulated into the weight value of the label value to generate a label value identification result containing the labeling result and the weight value. 4.The multi-modal large model based management method of multimedia material label according to claim 3, wherein, When the labeling scheme is based on keyword matching, the text segment is traversed to match the preset keyword list, including: Obtain the preset keyword list corresponding to the target label value, which contains multiple feature words related to the label value; Traverse the text segment of each hierarchical node in the structured text information, and perform word segmentation processing to obtain multiple word units; Accurately match the word units with the feature words in the preset keyword list, record the position of the matched word units in the text segment and the corresponding hierarchical node identification; Statistically count the number of successful matches of each hierarchical node text segment, and perform weighted processing on the number of successful matches according to the importance coefficient of the hierarchical node, which is negatively related to the depth of the hierarchical node; According to the weighted number of successful matches, determine the keyword matching result of the label value, and when the weighted number exceeds the preset threshold, determine that the keyword matching labeling of the label value is successful. 5.The multi-modal large model based management method of multimedia material label according to claim 3, wherein, When the tagging scheme is based on an association rule, whether the text segment meets the preset logical condition is verified, including: Parses the preset association rule corresponding to the target label value, the association rule contains multiple condition items and logical relationships between the condition items, and the logical relationships include an and relationship, an or relationship, and a not relationship; Extracts a text segment related to the condition item in the structured text information, performs semantic analysis on the text segment, and determines whether the text segment meets the requirement of a single condition item; According to the logical relationship, the satisfaction results of the multiple condition items are combined to determine whether the association rule is established; When the logical relationship is an and relationship, all condition items meet the requirement; When the logical relationship is an or relationship, at least one condition item meets the requirement; When the logical relationship is a not relationship, a specified condition item does not meet the requirement; Records the judgment process and the text segment position of the association rule, and generates an association rule verification report; 6.The multi-modal large model based management method of multimedia material label according to claim 3, wherein, When the association rule is established, it is determined that the association rule tagging of the label value is successful. When the tagging scheme is based on semantic understanding, a semantic summary of the text segment is generated by a multi-modal large model and matched with the label value description, including: Input the text segment of the structured text information into the multi-modal large model, generate a semantic summary of the text segment by the text understanding module of the multi-modal large model, and the semantic summary contains the core theme and key information of the text segment; Obtain the description text of the target label value, and the description text contains the definition, application scenario, and feature description of the label value; Input the semantic summary and the label value description text into the semantic matching module of the multi-modal large model, calculate the semantic similarity between the two, and the semantic similarity is determined based on the word vector cosine distance and the semantic role overlap degree; When the semantic similarity exceeds a preset threshold, further analyze whether the semantic summary contains the key features in the label value description text by the multi-modal large model; 7.The multi-modal large model based management method of multimedia material label according to claim 1, wherein, If the key features are contained, it is determined that the semantic understanding tagging of the label value is successful, and the semantic similarity value is recorded as the basis for weight value calculation. Integrate the tagging results and weight values of each target label group, filter and sort the tagging results according to the attributes of the label group, and generate a final label set of the multimedia material, including: Collect the tagging results and corresponding weight values of all label values in each target label group, and establish a temporary label value set, the temporary set contains label value identification, tagging result state, and weight value; Analyze the attribute information of the target label group, the attribute information includes a must-tag attribute, a tolerance number attribute, and a priority attribute, the must-tag attribute indicates that at least one label value is required under the label group, the tolerance number attribute indicates the maximum number of label values that can be contained under the label group, and the priority attribute indicates the sorting priority of the label value; For the label group containing the must-tag attribute, if the number of label values with successful tagging result in the temporary set is zero, mark the label group as an abnormal state and record the abnormal information; For the label group containing the tolerance number attribute, the number of label values with successful tagging results in the temporary set is counted. If the number exceeds the tolerance number attribute value, the label values are sorted according to the priority attribute, the label values with high priority are arranged in front, the label values with the same priority are sorted according to the weight value from large to small, and the tolerance number of label values with high priority are selected and retained in the temporary set. The label values in the temporary set after processing of each target label group are integrated, and the repeated label values are removed to generate a final label set of the multimedia material, the final label set containing label value identifiers, belonging label groups and weight values. The method for processing the target label group, counting the number of label values with successful tagging results in the temporary set, and selecting the label values with high priority according to the priority attribute, includes the following steps: The tolerance number attribute value and the priority attribute value of each label value are read from the attribute information of the target label group, and the priority attribute value is used to represent the relative importance of the label value. The label values with successful tagging results in the temporary set are traversed, the priority attribute value and the weight value of each label value are extracted, and a sorting feature list is established. The label values in the sorting feature list are sorted according to the priority attribute value from high to low in the first level, and the label values with the same priority attribute value are classified into the same priority group. The label values in each priority group are sorted according to the weight value from large to small in the second level to generate the sorting result in the group. The priority groups after the first level sorting are arranged in order, and the label values in each priority group are arranged according to the sorting result in the group to form a complete label value sorting sequence. The first tolerance number of label values are selected from the label value sorting sequence in turn and retained in the temporary set, and the remaining label values are removed from the temporary set. 8.The multi-modal large model based management method of a multimedia material label according to claim 1, wherein, The tree-shaped label system and the tagging scheme are versioned and maintained, and the label system change information, the tagging scheme change information and the label result version information are recorded, including: A change request of the tree-shaped label system is received, and the change request contains operation instructions of adding a label group, deleting a label group, modifying an attribute of a label group, adding a label value or modifying an attribute of a label value. The change request is approved, it is verified whether the change operation conforms to the preset change rule, and after the approval, the change operation is performed and a new label system version is generated, and the version identifier, the change time, the change content and the change executor information are recorded, and the change rule includes the label group naming specification, the hierarchical structure constraint and the attribute value range limitation. A change request of the tagging scheme is received, and the change request contains operation instructions of adding a tagging scheme, modifying a tagging scheme logic or adjusting a weight value of a tagging scheme. The change request of the tagging scheme is verified for validity, it is verified whether the modified tagging scheme logic has syntax error or logic conflict, and after the verification, the change operation is performed and a new tagging scheme version is generated, and the version identifier, the change time, the change content and the change executor information are recorded. When the tree label system or the marking scheme is updated, the affected multimedia material is identified, the label value identification and integration processing of the affected material is re-executed, a new label result version is generated, and the label result version identification, the associated label system version, the marking scheme version and the generation time are recorded.

9. A multi-modal large model based multimedia material label management system, characterized by, The application relates to a multimedia material label management method based on a multi-modal large model, and comprises a processor and a memory, wherein the memory is connected with the processor, the memory is used for storing programs, instructions or codes, and the processor is used for executing the programs, instructions or codes in the memory to realize the multimedia material label management method based on the multi-modal large model in any one of claims 1-8. The application relates to a multimedia material label management method based on a multi-modal large model, and comprises a processor and a memory, wherein the memory is connected with the processor, the memory is used for storing programs, instructions or codes, and the processor is used for executing the programs, instructions or codes in the memory to realize the multimedia material label management method based on the multi-modal large model in any one of claims 1-8.

Images