Metadata processing method, computer equipment and readable storage medium

Abstract

The invention discloses a metadata processing method, computer equipment and a readable storage medium. The method comprises the steps of obtaining an unstructured text; according to the unstructured text and a processing instruction corresponding to metadata processing, generating a first model cue word, and inputting the first model cue word into a trained large language model; and processing the unstructured text through the large language model, and outputting metadata information of general attributes and metadata information of sensitive attributes in the unstructured text. In this way, the efficiency and the intelligent level of metadata information extraction can be improved. Moreover, besides the metadata with the general attributes, the metadata with the sensitive attributes can be extracted, and management of sensitive information is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a metadata processing method, a computer device and a computer-readable storage medium. Background Art

[0002] With the deepening of digital transformation and AI applications, enterprises have begun to pay more and more attention to data analysis, mining, and the development and utilization of data resources. For example, knowledge graphs and AIGC applications have become the basis for enterprises to provide data services internally and externally. The development of these applications requires enterprises to store a large number of different types of data, such as pictures, audio, video, design documents, and materials. Such data is often stored in various systems of the enterprise and is in a decentralized state. It contains more complex information but has a relatively low value density, making it difficult to simply and effectively associate it. In order to make full use of such information, it is necessary to establish a complete metadata management system to manage structured and unstructured data, thereby eliminating islands between data and enabling data to be fully circulated and shared within the enterprise.

[0003] In metadata management, the structured metadata model can be composed of business metadata, technical metadata, and management metadata. However, there is no unified standard for metadata of unstructured data. In the past, the method was to extract different types of unstructured data separately. For example, audio was first converted to text and then extracted through regular expressions, and videos were manually labeled and video parameter parsing tools (Mediainfo) were used to extract metadata. Such methods often require a lot of manpower to extract attribute features, which is time-consuming and labor-intensive and cannot guarantee that all valuable data is fully collected. At the same time, it is also heavily dependent on the prior knowledge and background information of metadata managers, and the degree of standardization is relatively low. Therefore, the existing unstructured metadata processing has the problem of low efficiency. Summary of the invention

[0004] The purpose of the present application is to provide a metadata processing method, a computer device and a computer-readable storage medium, which can improve the efficiency of metadata processing of unstructured text.

[0005] To achieve the above objectives:

[0006] In a first aspect, an embodiment of the present application provides a metadata processing method, the method comprising:

[0007] Access unstructured text;

[0008] Generate a first model prompt word according to the unstructured text and a processing instruction corresponding to the metadata processing, and input the first model prompt word into a trained large language model;

[0009] The unstructured text is processed by the large language model to output metadata information of common attributes and metadata information of sensitive attributes in the unstructured text.

[0010] In one embodiment, the processing of the unstructured text by the large language model to output metadata information of common attributes and metadata of sensitive attributes in the unstructured text includes:

[0011] Performing text information extraction processing on the unstructured text by using the large language model to extract metadata information corresponding to preset attributes in the unstructured text; the preset attributes include general attributes and sensitive attributes;

[0012] The unstructured text is subjected to text reasoning processing by means of the large language model to complete the metadata information of the preset attributes that are missing.

[0013] In one embodiment, the method further comprises:

[0014] Acquire a sample set, the sample set including a first sample set; the samples in the first sample set include a first unstructured text containing sensitive data, and sensitive data information of the first unstructured text;

[0015] Generate a second model prompt word according to the sample of the sample set; for the sample of the first sample set, the second model prompt word is generated according to the first unstructured text, the sensitive data information and the processing instruction corresponding to the metadata processing;

[0016] The second model prompt word is input into the large language model to be trained, and the large language model is trained until the training end condition is met, thereby obtaining a trained large language model. In one embodiment, the sensitive data information includes at least two types of sensitive information;

[0017] The method of obtaining the first sample set includes:

[0018] Acquire an original first sample set through a first channel; the samples in the original first sample set include a first unstructured text containing sensitive data, and basic sensitive information of the first unstructured text;

[0019] Generate a third model prompt word according to the first unstructured text, the basic sensitive information and the processing instruction corresponding to the sensitive information processing;

[0020] The third model prompt word is input into a universal large language model, and is processed by the universal large language model to output sensitive data information of the first unstructured text; the information types of the sensitive data information are more than the information types of the basic sensitive information.

[0021] In one embodiment, the basic sensitive information includes sensitive data classification; the sensitive data classification is used to indicate whether the first unstructured text includes sensitive data and / or a category of sensitive data;

[0022] The sensitive data information includes: sensitive data vocabulary, classification of the sensitive data vocabulary, at least one of compliance security prompts and data security levels, and the sensitive data classification.

[0023] In one embodiment, the sample set further includes a second sample set, and the method further includes:

[0024] Acquiring the second sample set through a second channel; the samples in the second sample set include second unstructured text;

[0025] For the samples of the second sample set, the second model prompt words are generated according to the processing instructions corresponding to the second unstructured text and metadata processing.

[0026] In one embodiment, the method further comprises:

[0027] Obtain an initial general large language model;

[0028] In response to a fine-tuning instruction, the large language model is fine-tuned to obtain a large language model to be trained; the fine-tuning instruction is used to instruct the execution of freezing the weight parameters of each layer of the general large language model, defining a loss function and an optimizer, adding a specific model layer to the general large language model, combining the general large language model with the newly added model layer, compiling the combined model, and defining at least one of model parameters.

[0029] In one embodiment, the method further comprises:

[0030] Integrating the metadata information of the common attributes and the metadata information of the sensitive attributes in the unstructured text to obtain metadata of the unstructured text;

[0031] The metadata of the unstructured text is stored in a structured manner.

[0032] In a second aspect, an embodiment of the present application provides a device, comprising: a processor and a memory storing a computer program, and when the processor runs the computer program, the steps of the above-mentioned metadata processing method are implemented.

[0033] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned metadata processing method are implemented.

[0034] The metadata processing method, computer device, and computer-readable storage medium provided in the embodiments of the present application use a large language model to extract metadata from unstructured texts without pre-defining relevant pattern matching information, and can avoid the problem of manually filling, extracting, and summarizing metadata information, thereby improving the efficiency and intelligence level of metadata information extraction. In addition, in addition to being able to extract metadata of general attributes, metadata of sensitive attributes can also be extracted, which facilitates the management of sensitive information and facilitates the subsequent application of flexible application strategies to sensitive information in downstream applications, such as desensitizing sensitive information in downstream applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a metadata processing method provided by an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of a flow chart of another metadata processing method provided by an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of a flow chart of the steps of training a large language model provided by an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of a process for obtaining a first sample set provided by an embodiment of the present invention;

[0039] Figure 5 A schematic flow chart of another step of training a large language model provided by an embodiment of the present invention;

[0040] Figure 6 A schematic diagram of a process of fine-tuning a large language model in one embodiment;

[0041] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0043] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0044] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in this article, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this article, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "at the time of..." or "when..." or "in response to determination". Furthermore, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising", "including" indicate that there are described features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Thus, “A, B, or C” or “A, B and / or C” means “any of the following: A; B; C; A and B; A and C; B and C; A, B, and C.” An exception to this definition will occur only when a combination of elements, functions, steps, or operations are inherently mutually exclusive in some manner.

[0045] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0046] It should be noted that, in this article, step codes such as S101, S102, etc. are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence. When implementing the step, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the scope of protection of this application.

[0047] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0049] See also Figure 1 , is a metadata processing method provided in an embodiment of the present application. The method can be executed by a computer device provided in an embodiment of the present application. The computer device can be implemented in software and / or hardware. In this embodiment, the execution subject of the method is a computer device as an example. The method provided in this embodiment includes:

[0050] Step S101: Obtain unstructured text data.

[0051] Among them, unstructured text refers to text data that does not have a fixed format or predefined data model to organize information. In contrast, structured text, such as a table in a database, has a clear meaning for each column and each row follows the same structure.

[0052] Step S102: Generate a first model prompt word according to the unstructured text and the processing instruction corresponding to the metadata processing, and input the first model prompt word into the trained large language model.

[0053] When processing unstructured text, traditional unstructured data metadata processing methods such as rule system extraction, keyword matching, and topic modeling can be used to extract useful information from unstructured text and convert it into structured data for more in-depth analysis and mining. The rule system uses predefined rules and patterns, such as regular expressions and keyword lists, to extract metadata. This method often has the characteristics of complex rule formulation, poor flexibility, and high maintenance costs. Keyword matching extracts metadata information by pre-finding and determining keywords. This method has the problems of low accuracy and limited context understanding. Topic modeling (LDA) analyzes the potential topics in the text and assigns the text content to various topics. It often has the problem of difficult results interpretation and inability to understand the information in a fine-grained manner.

[0054] In addition, traditional unstructured data metadata processing methods all require the use of specific methods to solve the metadata extraction problems of different types of unstructured data, and lack scalability.

[0055] In view of the defects of metadata processing of traditional unstructured data, this application uses a large language model to extract metadata from unstructured text data. Large language model (LLM) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language model can handle a variety of natural language tasks, such as text classification, question and answer, dialogue, etc., and is an important path to artificial intelligence.

[0056] Model prompts refer to keywords, phrases, or sentences provided when interacting with a large language model to guide the model to generate a desired response or complete a specific task. Model prompts are critical to the quality of the model's output because they can affect the model's focus, thinking path, and the text content that is ultimately generated.

[0057] Taking the metadata processing of this application as an example, the corresponding model prompt words are extracted according to the unstructured text and metadata to generate model input data, which can enable the large language model to understand the requirements and improve the accuracy of the output.

[0058] The processing instruction is the model task. For example, the processing instruction corresponding to metadata processing may be “Please extract metadata of the following text”, and the model prompt may be “Please extract metadata of the following text: ”.

[0059] In other embodiments, in order to enable a large language model to unify the output format, unstructured text, processing instructions corresponding to metadata processing, and output format requirements may also be used as model prompts. The output format requirements may include attribute categories corresponding to metadata, such as title, content description, creator, data source, data format, data manager, security and compliance, etc. It is understandable that the attribute categories included in the output format requirements should cover the attribute categories required for metadata processing, including general attribute categories and sensitive attribute categories.

[0060] Step 103: Process the unstructured text by using a large language model, and output metadata information of common attributes and metadata information of sensitive attributes in the unstructured text.

[0061] Among them, common attributes refer to those general features that are commonly found in data sets and are usually used to describe or identify data subjects. These attributes do not directly involve personal privacy or sensitive information, so they can usually be shared and used in a relatively broad range. In one embodiment, for example, title, content description, creator, data source, data format, and data manager belong to the category of common attributes.

[0062] Among them, sensitive attribute categories refer to those characteristics that involve personal privacy, may cause discrimination or prejudice, or are protected by law. These attributes need to be handled and shared with special care to ensure personal privacy and data security. In one embodiment, for example, security and compliance belong to the sensitive attribute category.

[0063] Large language models are one of the important breakthroughs in the field of artificial intelligence in recent years. Based on deep learning technology, large language models generate models with powerful natural language understanding and generation capabilities through training on massive data. In this embodiment, a large language model is used to process unstructured text through the large language model to output metadata information of each common attribute in the unstructured text and metadata of each sensitive attribute.

[0064] In this embodiment, the metadata of unstructured text is extracted using a large language model without pre-defining relevant pattern matching information, which can avoid the problem of manually filling, extracting, and summarizing metadata information, thereby improving the efficiency and intelligence level of metadata information extraction. In addition, in addition to being able to extract metadata of general attributes, metadata of sensitive attributes can also be extracted, which facilitates the management of sensitive information and facilitates the subsequent application of flexible application strategies to sensitive information in downstream applications, such as desensitizing sensitive information in downstream applications.

[0065] In another embodiment, unstructured text is processed by a large language model to output metadata information of each common attribute and metadata of each sensitive attribute in the unstructured text, including: performing text information extraction processing on the unstructured text by a large language model to extract metadata information corresponding to preset attributes in the unstructured text; the preset attributes include common attributes and sensitive attributes; performing text reasoning processing on the unstructured text by a large language model to complete metadata information of preset attributes that are missing information.

[0066] The preset attributes may be the attribute categories listed in the output format requirements of the model prompt words, or may be the attribute categories included in the output format requirements learned during the large language model training process. The processing of the large language model includes text extraction processing and text reasoning processing. Text extraction processing refers to directly extracting metadata information corresponding to the preset attributes from unstructured text. For example, if the unstructured text includes that someone is 20 years old and his occupation is a teacher, metadata such as name, age and occupation can be directly extracted.

[0067] In some embodiments, there may be content of preset attributes that are not directly recorded in the unstructured text and cannot be directly obtained through text information extraction processing. For example, a preset attribute is content description, which is the summary content of the unstructured text, and a text usually does not directly record the content of the content description. Therefore, it is difficult to extract the metadata corresponding to the content description through text extraction processing. At this time, the large language model is used to perform text reasoning processing on the unstructured text to complete the metadata information of the attribute of the content description of the unstructured text. For another example, a preset attribute is that sensitive information is a security level, which involves the security level of the data describing the unstructured text. The security level may include public disclosure, internal disclosure, internal control, and strict internal control. If the security level is not directly recorded in the unstructured text, it is difficult to directly extract the metadata corresponding to the security level. At this time, the large language model is used to perform text reasoning processing on the unstructured text to complete the metadata information of the attribute of the security level of the unstructured text.

[0068] In this embodiment, by using a large language model to perform text information extraction and text reasoning processing on unstructured text, metadata information of preset attributes of the unstructured text can be accurately and completely processed.

[0069] In another embodiment, Figure 2 As shown, a metadata processing method includes the following steps:

[0070] Step S201, obtaining unstructured text.

[0071] Step S202: Generate first model prompt words according to the unstructured text and the processing instructions corresponding to the metadata processing, and input the first model prompt words into the trained large language model.

[0072] Step S203, performing text information extraction processing on the unstructured text through the large language model, extracting metadata information corresponding to preset attributes in the unstructured text; the preset attributes include general attributes and sensitive attributes.

[0073] In one embodiment, the unstructured text to be extracted metadata is processed into a data structure acceptable to the model, including instructions and text. After the text is input into the model, the metadata to be extracted is output, and the extracted content includes basic information, title / name, title of data or document, creator / author (person or team who created or wrote the data), creation / modification date (date and time when the data was created or modified), data structure and format, data type, index and retrieval. When extracting metadata, the source and origin of the data automatically default to the collected source system information.

[0074] Step S204: Perform text reasoning on the unstructured text using a large language model to complete metadata information of missing preset attributes.

[0075] In one embodiment, since some metadata attributes cannot be directly obtained from the text, a large language model needs to be generated. First, the unstructured text to be used for metadata extraction is processed into a data structure acceptable to the model, and then input into the large language model to complete the content description (summary content of unstructured data), security and compliance related information, so as to enhance such metadata attributes.

[0076] Step S205 , integrating metadata information of common attributes and metadata information of sensitive attributes in the unstructured text to obtain metadata of the unstructured text.

[0077] Specifically, the integration method includes sorting and merging. Sorting can specifically be sorting the order of each preset attribute so that each preset attribute is output and presented in order. Merging can specifically be merging common attributes and sensitive attributes into a data table. The metadata information of the common attributes extracted and completed and the metadata of each sensitive attribute are sorted and merged to obtain the metadata of the unstructured text.

[0078] Step S206: storing the metadata of the unstructured text in a structured manner.

[0079] Specifically, the structured metadata information of unstructured text is stored in the data center to facilitate subsequent compliance risk analysis, data lineage analysis, metadata permission management and other applications.

[0080] The metadata processing method of this embodiment uses a large language model to extract metadata from unstructured text without pre-defining relevant pattern matching information, which can avoid the problem of manually filling, extracting, and summarizing metadata information, and can improve the efficiency and intelligence level of metadata information extraction. Furthermore, this method can innovatively manage metadata information of unstructured data, facilitating the development of subsequent applications.

[0081] In another embodiment, a metadata processing method further includes the step of training a large language model. Figure 3 As shown, the step includes:

[0082] Step S301: acquiring a sample set, the sample set including a first sample set; the samples in the first sample set include a first unstructured text containing sensitive data, and sensitive data information of the first unstructured text.

[0083] The sensitive data information is used to describe the type and / or content of the sensitive data in the first unstructured text. For example, the sensitive data information may include whether sensitive data is involved, which words are involved in sensitive data, the classification of sensitive data words, the security level of sensitive data, etc.

[0084] The sensitive data information of the first unstructured sample may be obtained by manual pre-processing or by using a general large language model.

[0085] Step S302, generating a second model prompt word according to the samples of the sample set; wherein, for the samples of the first sample set, the second model prompt word is generated according to the processing instructions corresponding to the first unstructured text, sensitive data information and metadata processing.

[0086] In this embodiment, a second model prompt word is generated according to processing instructions corresponding to the first unstructured text, sensitive data information and metadata processing, so that the large language model can understand the sensitive data information included in the first unstructured text and the training requirements of the large language model, thereby improving the accuracy of model training.

[0087] Step S303: input the second model prompt word into the large language model to be trained, and train the large language model until the training end condition is met, thereby obtaining the trained large language model.

[0088] In this embodiment, the second model prompt words are used to train the large language model to be trained. Since the second model prompt words include sensitive data information of the first unstructured text, the ability of the large language model to understand and identify sensitive data information in the first unstructured text can be enhanced during the training process, so that the trained large language model is enhanced in identifying and processing sensitive information, thereby improving the ability to process metadata of sensitive attributes in unstructured text.

[0089] Specifically, the training end condition may include the number of iterations.

[0090] Among them, the sensitive data information in each first unstructured text in the first sample set can be obtained by manual annotation, but this method will inevitably increase the workload and has low efficiency.

[0091] To address this problem, one embodiment provides a more efficient method, which only requires a small amount of manual annotation of basic sensitive information to supplement and improve the sensitive data information of the first unstructured text.

[0092] like Figure 4 As shown, in one embodiment, the method of obtaining the first sample set includes:

[0093] Step S401: obtaining an original first sample set through a first channel; the samples in the original first sample set include a first unstructured text containing sensitive data, and basic sensitive information of the first unstructured text.

[0094] Specifically, the first channel is a channel that can obtain texts containing sensitive information, such as some human resource management platforms containing sensitive information.

[0095] The basic sensitive information may be obtained by manual pre-annotation, including the most basic types of sensitive information. In one embodiment, the basic sensitive information includes sensitive data classification; the sensitive data classification is used to indicate whether the first unstructured text includes sensitive data and / or the category of sensitive data.

[0096] Step S402: Generate a third model prompt word according to the first unstructured text, the basic sensitive information and the processing instructions corresponding to the sensitive information processing.

[0097] The processing instructions corresponding to the sensitive information are used to instruct the large language model to extract and identify the sensitive information in the first unstructured text. For example, the processing instructions corresponding to the sensitive information may be "Please identify the security level and compliance risk warning of the data from the following text", or "Please identify the sensitive words and categories of sensitive words in the data from the following text".

[0098] Step S403: input the third model prompt word into the general large language model, process it through the general large language model, and output sensitive data information of the first unstructured text; the information types of the sensitive data information are more than the information types of the basic sensitive information.

[0099] In this embodiment, a large language model is used to extract sensitive data information of the first unstructured text, so that only simple basic sensitive information is needed to extract sensitive data information of the first unstructured text with richer information types.

[0100] In one embodiment, the sensitive data information includes: sensitive data vocabulary, classification of sensitive data vocabulary, at least one of compliance security prompts and data security levels, and sensitive data classification.

[0101] In one embodiment, the format of the third smash prompt word is as follows: "Task instruction: {""}, input text: {""}, output: {""}", for example, "Task instruction {"Please identify the data security level and compliance risk warning information from the following text"}, input text: {"Text involving personal health information, financial information and medical data"}, output: {"Strict internal control", "Sensitive data exists, such as personal health information and financial information, and there is a compliance risk"}.

[0102] In this embodiment, the third model prompt word is generated according to the first unstructured text, the basic sensitive information and the processing instructions corresponding to the sensitive information, so that the large language model can understand the sensitive data information included in the first unstructured text and the training requirements of the large language model for sensitive information recognition, thereby improving the ability of the large language model in sensitive information recognition. In this way, human operations are reduced, and the sensitive data information of the first unstructured text can be extracted with only simple basic sensitive information.

[0103] In another embodiment, the step of training a large language model, such as Figure 5 As shown, including:

[0104] Step S501, obtaining a first sample set; the samples in the first sample set include a first unstructured text containing sensitive data, and sensitive data information of the first unstructured text.

[0105] Specifically, the process of obtaining the first sample set has been recorded in the previous embodiment and will not be repeated here.

[0106] Step S502: Processing instructions corresponding to the first unstructured text, the sensitive data information and the metadata to generate a second model prompt word.

[0107] Specifically, the process of generating the second model prompt word according to the processing instructions corresponding to the first unstructured text, sensitive data information and metadata processing has been recorded in the previous embodiments and will not be repeated here.

[0108] Step S503: Acquire a second sample set through a second channel; the samples in the first sample set include a second unstructured text.

[0109] Specifically, the second channel is not limited to the channel for obtaining sensitive information. The second unstructured text in the second sample set is not limited to the text including sensitive data. For example, unstructured texts such as files in the ECM enterprise file management system and legal and regulatory documents of various countries are collected as the second unstructured text.

[0110] Step S504: Process the corresponding processing instructions according to the second unstructured text and the metadata to generate a second model prompt word.

[0111] For example, the model prompt word for the second unstructured text may be “Please extract metadata <second unstructured text> from the following text data”.

[0112] It can be understood that the above steps S501-S502 are for generating the second model prompt words for the first unstructured text, and the above steps S503-S504 are for generating the second model prompt words for the second unstructured text.

[0113] Step S505: input the second model prompt word into the large language model to be trained, and train the large language model until the training end condition is met, thereby obtaining the trained large language model.

[0114] In this embodiment, the large language model is trained using the first structured text that explicitly includes sensitive information and the second structured text that does not explicitly include sensitive information, so that the model has the ability to extract metadata information of sensitive attributes and general attributes. Among them, the second model prompt word corresponding to the first structured text that explicitly includes sensitive information includes sensitive data information, so that the trained large language model is enhanced in sensitive information recognition and processing, thereby improving the ability to process metadata of sensitive attributes in unstructured text.

[0115] In another embodiment, a method for obtaining a large language model to be trained includes: obtaining a general large language model; and responding to a fine-tuning instruction to fine-tune the large language model to obtain the large language model to be trained.

[0116] That is to say, the large language model to be trained in this embodiment is obtained by further fine-tuning the general large language model. The general large language model has generated corresponding model weights through early training and has rich language knowledge and patterns. In this embodiment, fine-tuning is performed on the basis of the general large language model to improve the efficiency of model training.

[0117] Specifically, the fine-tuning instructions include: instructing execution to freeze the weight parameters of each layer of the general large language model, define a loss function and an optimizer, add a specific model layer to the general large language model, combine the general large language model with the newly added model layer, compile the combined model, and define at least one of the model parameters.

[0118] In one embodiment, if Figure 6 As shown, the fine-tuning process includes:

[0119] Step 1: Load the initial general large language model.

[0120] Step 2: When traversing each layer of the model, freeze the weight parameters of each layer of the original model to prevent the fine-tuning process from losing the original language characteristics or model characteristics.

[0121] Step 3: Define the loss function and optimizer, and select the appropriate loss function and optimizer. The loss function can be cross entropy loss, mean square error, etc. The optimizer can be Adam (Adaptive Moment Estimation), SGD (Stochastic Gradient Descent), Adagrad (Adaptive Gradient Algorithm), etc.

[0122] Step 4: Add specific model layers to the big language model, combine the pre-trained model with the newly added model layers, and compile the entire model.

[0123] For example, modify the dimension of the output layer, modify the dimension of the input layer, or add a fully connected layer, a convolutional layer, etc. By adding specific layers, it will help the large language model to generalize the metadata attributes of this type of specific unstructured data.

[0124] Step 5: Define the model parameters, including setting the learning rate, batch size, number of training rounds, etc.

[0125] Based on the same inventive concept as the above embodiment, the above embodiment is described in detail below through a specific application scenario.

[0126] In one embodiment, a first unstructured text is first obtained from a first channel (e.g., a MySQL database), as well as unstructured text data such as documents and legal and regulatory documents of various countries from a second channel (e.g., an ECM enterprise document management system). For the first unstructured text and the second unstructured text, the above steps S501 to S505 are executed to obtain a trained large language model.

[0127] For the unstructured text to be processed, the trained large language model is used to execute the above steps S201-S206, and the metadata of the unstructured text is extracted using the large language model.

[0128] In one embodiment, the unstructured data metadata attributes may include:

[0129] (1) Basic information: Title / name, title of the data or document, content description (summary content of unstructured data), creator / author (the person or team who created or wrote the data), creation / modification date (the date and time when the data was created or modified).

[0130] (2) Data source and provenance: Source (source of unstructured data).

[0131] (3) Data structure and format: data format (data storage format, such as PDF, JPEG, word, etc.), data type (data type, such as text, image, video, etc.).

[0132] (4) Data managers: business owner (business department that owns the data), data user (business or technical department that uses the data), and data maintainer (technical department that maintains the data).

[0133] (5) Indexing and retrieval: Keywords / tags (keywords, tags or topics used to retrieve data).

[0134] (6) Security and compliance: security level (data security level, such as open to the public, open to the public internally, internally controlled, and strictly controlled internally), compliance risk description (data compliance status, whether there is sensitive information, and whether there is a violation of compliance requirements of laws and regulations, such as GDPR, HIPAA, etc.).

[0135] It can be seen that the above metadata attributes include general attributes and sensitive attributes. This application uses a large language model to extract metadata from unstructured text without pre-defining relevant pattern matching information, which can avoid the problem of manually filling, extracting, and summarizing metadata information, and can improve the efficiency and intelligence level of metadata information extraction. Furthermore, this method can innovatively manage the metadata information of unstructured data, facilitating the development of subsequent applications.

[0136] In one embodiment, the metadata processing method of the present application can obtain first unstructured data and second unstructured data in the field of engineering machinery from relevant channels, and train a large language model, so that the metadata processing method of the present application can be used in the field of engineering machinery, thereby improving the efficiency of metadata extraction of unstructured text in the field of engineering machinery.

[0137] Based on the same inventive concept as the above embodiments, an embodiment of the present invention provides a computer device, such as Figure 7 As shown, the device includes: a processor 310 and a memory 311 storing a computer program; wherein, Figure 7 The processor 310 shown in the figure is not used to indicate that the number of the processor 310 is one, but is only used to indicate the positional relationship of the processor 310 relative to other devices. In actual applications, the number of the processor 310 may be one or more; similarly, Figure 7 The memory 311 shown in the figure has the same meaning, that is, it is only used to refer to the position relationship of the memory 311 relative to other devices. In practical applications, the number of memories 311 can be one or more. When the processor 310 runs the computer program, the metadata processing method applied to the above device is implemented.

[0138] The device may also include: at least one network interface 312. The various components in the device are coupled together via a bus system 313. It is understood that the bus system 313 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 313 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as bus system 313.

[0139] The memory 311 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. The nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 311 described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable types of memories.

[0140] The memory 311 in the embodiment of the present invention is used to store various types of data to support the operation of the device. Examples of these data include: any computer program used to operate on the device, such as an operating system and an application; contact data; phone book data; messages; pictures; videos, etc. Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program may include various applications, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. Here, the program that implements the method of the embodiment of the present invention may be included in the application program.

[0141] Based on the same inventive concept as the above-mentioned embodiment, this embodiment further provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium. The computer-readable storage medium may be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); or it may be various devices including one or any combination of the above-mentioned memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc. When the computer program stored in the computer-readable storage medium is executed by the processor, the above-mentioned metadata processing method is implemented. For the specific steps implemented when the computer program is executed by the processor, please refer to Figure 1 The description of the illustrated embodiment will not be repeated here.

[0142] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] In this document, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion of elements other than those listed and may also include additional elements not expressly listed.

[0144] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A metadata processing method, characterized in that: include: Access unstructured text; Generate a first model prompt word according to the unstructured text and a processing instruction corresponding to the metadata processing, and input the first model prompt word into a trained large language model; The unstructured text is processed by the large language model to output metadata information of common attributes and metadata information of sensitive attributes in the unstructured text.

2. The method according to claim 1, characterized in that The processing of the unstructured text by the large language model to output metadata information of common attributes and metadata of sensitive attributes in the unstructured text includes: Performing text information extraction processing on the unstructured text by using the large language model to extract metadata information corresponding to preset attributes in the unstructured text; the preset attributes include general attributes and sensitive attributes; The unstructured text is subjected to text reasoning processing by means of the large language model to complete the metadata information of the preset attributes that are missing.

3. The method according to claim 1, characterized in that The method further comprises: Acquire a sample set, the sample set including a first sample set; the samples in the first sample set include a first unstructured text containing sensitive data, and sensitive data information of the first unstructured text; Generate a second model prompt word according to the sample of the sample set; wherein, for the sample of the first sample set, the second model prompt word is generated according to the first unstructured text, the sensitive data information and the processing instruction corresponding to the metadata processing; The second model prompt word is input into the large language model to be trained, and the large language model is trained until a training end condition is met, thereby obtaining a trained large language model.

4. The method according to claim 3, characterized in that The sensitive data information includes at least two types of sensitive information; The method of obtaining the first sample set includes: Acquire an original first sample set through a first channel; the samples in the original first sample set include a first unstructured text containing sensitive data, and basic sensitive information of the first unstructured text; Generate a third model prompt word according to the first unstructured text, the basic sensitive information and the processing instruction corresponding to the sensitive information processing; The third model prompt word is input into a universal large language model, and is processed by the universal large language model to output sensitive data information of the first unstructured text; the information types of the sensitive data information are more than the information types of the basic sensitive information.

5. The method according to claim 4, characterized in that The basic sensitive information includes sensitive data classification; the sensitive data classification is used to indicate whether the first unstructured text includes sensitive data and / or a category of sensitive data; The sensitive data information includes: sensitive data vocabulary, classification of the sensitive data vocabulary, at least one of compliance security prompts and data security levels, and the sensitive data classification.

6. The method according to claim 3, characterized in that The sample set also includes a second sample set, and the method further includes: Acquiring the second sample set through a second channel; the samples in the second sample set include second unstructured text; For the samples of the second sample set, the second model prompt words are generated according to the processing instructions corresponding to the second unstructured text and metadata processing.

7. The method according to claim 3 or 6, characterized in that: The method further comprises: Obtain an initial general large language model; In response to a fine-tuning instruction, the large language model is fine-tuned to obtain a large language model to be trained; the fine-tuning instruction is used to instruct the execution of freezing the weight parameters of each layer of the general large language model, defining a loss function and an optimizer, adding a specific model layer to the general large language model, combining the general large language model with the newly added model layer, compiling the combined model, and defining at least one of model parameters.

8. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Integrating the metadata information of the common attributes and the metadata information of the sensitive attributes in the unstructured text to obtain metadata of the unstructured text; The metadata of the unstructured text is stored in a structured manner.

9. A computer device, characterized in that: include: A processor and a memory storing a computer program, wherein when the processor runs the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images