Metadata processing method based on large model and electronic equipment

Abstract

The invention provides a metadata processing method based on a large model and electronic equipment. The method and the electronic equipment are used for improving the universality of metadata standards. The method comprises the steps of obtaining directory content data in a data directory library, performing feature extraction on the directory content data to obtain data structure features, and extracting at least one piece of metadata from the directory content data according to the data structure features; inputting the at least one piece of metadata into a metadata reference model, and performing feature scoring on the at least one piece of metadata based on the metadata reference model to obtain scores corresponding to the at least one piece of metadata; and determining target metadata according to the score corresponding to the at least one piece of metadata, wherein the target metadata is a metadata standard in the at least one piece of metadata. The metadata corresponding to each data directory in the data directory library is scored by adopting the large model subjected to fine tuning training, so that a uniform metadata standard can be constructed for each data directory in the data directory library according to the score, and the universality is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a metadata processing method based on a large model and an electronic device. BACKGROUND

[0002] Data exchange in the automotive industry, as a core link of modern automotive industry, is of great significance for realizing vehicle intelligence, improving production efficiency and optimizing user experience. However, the automotive industry still faces many challenges in data exchange. For example, data security and privacy protection, complexity and difficulty of data exchange.

[0003] In view of the current problems of automotive industry data, by formulating a unified metadata standard, the standardized description and exchange of automotive industry data can be largely realized, and the quality and usability of data can be improved. The standardized metadata set not only promotes data interconnection between various parties in the automotive data element circulation link, but also supports higher level data analysis and mining, improves the realization of automotive data element value, and promotes data circulation. At the same time, after the standardization of metadata, it can be combined with data security, privacy protection and other regulatory requirements to ensure the legality and compliance of data in the exchange and use process.

[0004] At present, the establishment of metadata standards is mostly through the establishment of standards by institutions and enterprises together to build a metadata set. However, this way constructs the intersection of metadata standards formulated by institutions and enterprises, and the applicable scene is single and not easy to expand. SUMMARY

[0005] The present application provides a metadata processing method based on a large model and an electronic device to improve the universality of metadata standards.

[0006] In a first aspect, the present application provides a metadata processing method based on a large model, which comprises:

[0007] Obtaining directory content data in a data directory library;

[0008] Performing feature extraction on the directory content data to obtain data structure features, and extracting at least one metadata from the directory content data according to the data structure features;

[0009] Inputting the at least one metadata into a metadata benchmark model, performing feature scoring on the at least one metadata based on the metadata benchmark model to obtain the respective scores of the at least one metadata; the metadata benchmark model is a large model trained by fine-tuning;

[0010] According to the score corresponding to each of the at least one metadata, a target metadata is determined, the target metadata being a metadata standard in the at least one metadata.

[0011] Since the embodiments of the present application directly extract features of each data catalog in the data catalog library to obtain metadata of each data catalog in the data catalog library, score the metadata extracted from the data catalog library based on the metadata benchmark model, and then select metadata with stronger universality from the plurality of metadata as the metadata standard between each data catalog in the data catalog library, the embodiments of the present application can obtain the metadata standard adapted to the plurality of data catalogs in the data catalog library, no longer limit the number of participating data catalogs, reduce the difficulty of data exchange, thereby improving the universality of the metadata standard, and further realizing the standardized description and exchange of data, and improving the quality and availability of data.

[0012] An optional embodiment is that, before the at least one metadata is input into the metadata benchmark model, the method further comprises:

[0013] Obtaining a fine-tuning data set, the fine-tuning data set comprising a data flow transfer graph and market data corresponding to each data catalog in the data catalog library; the market data corresponding to one data catalog comprises evaluation information, usage information, and pricing information of the one data catalog;

[0014] According to a fine-tuning algorithm and the fine-tuning data set, the metadata benchmark model is updated.

[0015] Since the embodiments of the present application optimize the metadata benchmark model using the fine-tuning data set before using the metadata benchmark model, the accuracy of the metadata benchmark model is improved.

[0016] An optional embodiment is that, the obtaining of the fine-tuning data set comprises:

[0017] Feature extraction is performed on the catalog content data to obtain embedding vectors;

[0018] A data resource graph is constructed according to the correlation between the embedding vectors; the data resource graph is used to represent the correlation between each data catalog in the data catalog library;

[0019] Market data corresponding to the each data catalog is obtained, and the data flow transfer graph is obtained according to the market data and the data resource graph;

[0020] According to the data flow transfer graph and the market data, the fine-tuning data set is constructed.

[0021] Optionally, the method further comprises:

[0022] Optionally, the method further comprises:

[0023] Optionally, the method further comprises:

[0024] Optionally, the method further comprises:

[0025] Optionally, the method further comprises:

[0026] Optionally, the method further comprises:

[0027] Optionally, the method further comprises:

[0028] Optionally, the method further comprises:

[0029] Optionally, the method further comprises:

[0030] Optionally, the method further comprises:

[0031] Optionally, the method further comprises:

[0032] Optionally, the method further comprises:

[0033] Optionally, the method further comprises:

[0034] Optionally, the method further comprises:

[0035] obtain a sample data set, the sample data set comprising a plurality of standard metadata;

[0036] perform transfer learning on the large model according to the sample data set to obtain an initial metadata benchmark model.

[0037] In a second aspect, an embodiment of the present application provides a metadata processing apparatus based on a large model, comprising:

[0038] an obtaining unit configured to obtain directory content data in a data directory library;

[0039] an extracting unit configured to perform feature extraction on the directory content data to obtain data structure features, and extract at least one metadata from the directory content data according to the data structure features;

[0040] a scoring unit configured to input the at least one metadata into a metadata benchmark model, perform feature scoring on the at least one metadata based on the metadata benchmark model, and obtain respective scores of the at least one metadata; the metadata benchmark model is a large model trained through fine-tuning;

[0041] a determining unit configured to determine a target metadata according to the respective scores of the at least one metadata, the target metadata being a metadata standard in the at least one metadata.

[0042] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, the memory storing a computer program executable on the processor, and when the computer program is executed by the processor, the computer program implements the metadata processing method based on a large model as described in the first aspect.

[0043] In a fourth aspect, an embodiment of the present application further provides a computer storage medium, the computer storage medium storing computer program instructions, and when the instructions are executed on a computer, the computer program instructions cause the computer to execute the metadata processing method based on a large model as described in the first aspect.

[0044] The technical effects brought by any one of the implementation manners of the second aspect to the fourth aspect can refer to the technical effects brought by the corresponding implementation manners of the first aspect, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1 A flowchart of a metadata processing method based on a large model provided by an embodiment of the present application is shown in FIG. 1.

[0047] Figure 2 A metadata standard graph provided by an embodiment of the present application is shown in FIG. 2.

[0048] Figure 3 Another metadata standard graph provided by an embodiment of the present application is shown in FIG. 3.

[0049] Figure 4 Still another metadata standard graph provided by an embodiment of the present application is shown in FIG. 4.

[0050] Figure 5 A complete flowchart of a metadata processing method based on a large model provided by an embodiment of the present application is shown in FIG. 5.

[0051] Figure 6 A structural diagram of a metadata processing device based on a large model provided by an embodiment of the present application is shown in FIG. 6.

[0052] Figure 7 A structural block diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 7. DETAILED DESCRIPTION

[0053] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0054] Some terms appearing in the text will be explained below:

[0055] (1) Expert database: refers to the original database constructed by collecting various automobile data related association standards, national standards, etc. In the embodiments of the present application, the expert database includes but is not limited to: collected vehicle enterprise metadata graph, association released automobile data standard file, such as certificate standard file, vehicle networking data standard, national standard, etc.

[0056] (2) Market information database: refers to the basic database constructed by collecting automobile data market information. In the embodiments of the present application, the market information database includes but is not limited to data flow graph, market data. Among them, the market data includes but is not limited to usage record, business scenario description, user evaluation, pricing information and resource description of data, etc.

[0057] (3) Data directory: refers to a data resource directory. The directory composition includes but is not limited to directory address, directory description, directory introduction, metadata introduction and the like. In the embodiment of the application, the data directory can be an automobile data directory, and the data directory library can be an automobile data directory library for storing automobile data resource directories.

[0058] At present, the establishment mode of the metadata standard is mostly a standard formulated by an organization and an enterprise together, and a metadata set is constructed. However, this mode constructs the metadata intersection of the standards formulated by the organization and the enterprise, and the application scene is single and not easy to expand.

[0059] Based on the above problems, as shown in the formula (I), the embodiment of the application provides a metadata processing method based on a large model, including the following steps: Figure 1

[0060] S101: Obtain directory content data in a data directory library.

[0061] The data directory library can be an automobile data directory library, and the data directory in the data directory library can be an automobile data directory. The directory content data includes but is not limited to: directory address, directory description, directory introduction, metadata identification, metadata introduction.

[0062] S102: Feature extraction is performed on the directory content data to obtain data structure features, and at least one metadata is extracted from the directory content data according to the data structure features.

[0063] In some embodiments, the embodiment of the application can perform structured feature extraction on the directory content data to obtain the data structure features of the directory content data. The embodiment of the application can extract at least one metadata from the directory content data according to the extracted data structure features. The metadata includes but is not limited to: data type, column name, table structure.

[0064] The embodiment of the application can also perform semantic analysis on the directory content data through a natural language processing (NLP) technology to obtain a semantic analysis result. The NLP technology includes but is not limited to: RNN, LSTM, BERT, ChatGLM. For example, the embodiment of the application can perform semantic analysis on the metadata introduction information in the directory content data to obtain a semantic analysis result. The embodiment of the application can construct a metadata semantic graph according to the semantic analysis result of the metadata introduction information and the metadata. The metadata semantic graph is used to represent the mapping relationship between the metadata and the semantic analysis result.

[0065] ​S103: input the at least one metadata into the metadata benchmark model, perform feature scoring on the at least one metadata based on the metadata benchmark model, and obtain respective scores corresponding to the at least one metadata.

[0066] The metadata benchmark model is a large model trained through fine-tuning. In the embodiments of the present application, the large model can be an RNN, an LSTM, a BERT, a ChatGLM, or the like.

[0067] In some embodiments, when the metadata benchmark model is used to score the metadata, the metadata benchmark model needs to be optimized according to the usage of each data directory in the data directory library. In this way, the embodiments of the present application can obtain more accurate scores, and thus can ensure the universality of the target metadata determined subsequently.

[0068] For example, the embodiments of the present application can update the metadata benchmark model through the following steps:

[0069] In some embodiments, the embodiments of the present application can construct a fine-tuning data set according to the data in the market information library and the data directories in the data directory library.

[0070] The fine-tuning data set includes a data flow graph and market data corresponding to each data directory in the data directory library. The market data corresponding to one data directory includes evaluation information, pricing information, and usage information of the data directory. For example, the usage information in the market data includes but is not limited to usage records, business scenario descriptions, pricing information, and resource descriptions of the data directory. The evaluation information can be user evaluation when the data directory is used.

[0071] For example, the embodiments of the present application can perform feature extraction on the directory content data to obtain embedding vectors. For example, the embodiments of the present application can extract text features of the directory content data through NLP technology to generate context-related embedding vectors.

[0072] The embodiments of the present application can also construct a data resource graph according to the correlation between the embedding vectors. The data resource graph is used to represent the correlation between each data directory in the data directory library. For example, the embodiments of the present application can perform correlation analysis on the embedding vectors and construct a data resource graph based on the correlation between the embedding vectors.

[0073] This application embodiment can also acquire market data corresponding to each data catalog, and obtain a data flow map based on the market data and data resource map. For example, this application embodiment can acquire market data collected during the market circulation of each data catalog, and expand the data resource map based on the collected market data to obtain a data flow map. This application embodiment can construct a fine-tuned dataset based on the data flow map and market data. Furthermore, this application embodiment can also store the collected market data and data flow map in a market information database.

[0074] This application embodiment can also update the metadata baseline model based on the fine-tuning algorithm and the fine-tuning dataset. In this way, this application embodiment can utilize the fine-tuning dataset to fine-tune the metadata baseline model, thereby improving the accuracy of the metadata baseline model in metadata recognition tasks.

[0075] In this embodiment of the application, after updating the metadata benchmark model, at least one metadata can be input into the updated metadata benchmark model, and feature scoring can be performed on the at least one metadata based on the updated metadata benchmark model to obtain the score corresponding to each of the at least one metadata.

[0076] In some embodiments, the metadata baseline model can be obtained in the following ways:

[0077] Step 1: Obtain the sample dataset. The sample dataset includes several standard metadata.

[0078] In some embodiments, this application can extract multiple standard metadata from an automotive expert standard library to construct a sample dataset. The automotive expert standard library includes, but is not limited to: vehicle enterprise metadata maps, automotive data standard documents issued by associations, and national standards. The automotive data standard documents issued by associations include, but are not limited to: certificate of conformity standard documents and vehicle networking data standards.

[0079] For example, in the embodiments of this application, standard data in the automotive expert standard library can be used as standard metadata to obtain a sample dataset.

[0080] Step 2: Perform transfer learning on the large model based on the sample dataset to obtain the metadata baseline model.

[0081] In some embodiments, this application can use a sample dataset to train a large model and then transfer the trained model parameters to a metadata benchmark model. The metadata benchmark model has the ability to define metadata.

[0082] S104: Determine the target metadata based on the score corresponding to at least one metadata.

[0083] The target metadata is a metadata standard in at least one metadata.

[0084] In some embodiments, this application can perform semantic aggregation and classification on at least one metadata in the directory content data to obtain at least one metadata set. For example, this application can combine an attention model and an aggregation algorithm to perform semantic aggregation and classification on at least one metadata to obtain at least one metadata set. The metadata set consists of metadata with the same meaning.

[0085] As an example, embodiments of this application can perform semantic analysis on data related to at least one metadata in the directory content data to obtain semantic analysis results corresponding to each of the at least one metadata. Embodiments of this application can aggregate and classify the semantic analysis results to obtain at least one metadata set.

[0086] As another example, embodiments of this application can perform semantic aggregation and classification on the metadata semantic graph to obtain at least one metadata set.

[0087] After obtaining at least one metadata set, this application embodiment can further determine the target metadata for each metadata set based on the score corresponding to each metadata in each metadata set. For example, this application embodiment can use the metadata with the highest average score in the metadata set as the target metadata.

[0088] After determining the target metadata for each metadata set, this embodiment of the application can further construct a metadata standard graph based on the target metadata and other metadata in each metadata set. In this way, this embodiment of the application can perform data exchange between different data directories based on the metadata standard graph. Here, "other metadata" refers to the metadata in each metadata set excluding the target metadata, and the metadata standard graph is used to characterize the mapping relationship between the target metadata and other metadata in each metadata set. For example, if the metadata set includes metadata AN, YN, and CN, and AN has a score of 90, YN has a score of 80, and CN has a score of 100, then CN is used as the target metadata. Figure 2 As shown, in this embodiment of the application, a mapping relationship between CN and YN and AN can be constructed using CN as the metadata standard to obtain a metadata standard map.

[0089] In some embodiments, this application embodiment can also update the data directory in the data directory library. For example, when it is determined that a new data directory has been added to the data directory library, this application embodiment obtains the new directory content data of the new data directory. This application embodiment can perform feature extraction on the new directory content data to obtain at least one new metadata. The new metadata is the metadata of the new data directory. The process of obtaining the new metadata is the same as the process of obtaining metadata in step S102 above, and will not be repeated here.

[0090] When updating the data catalog, embodiments of this application can also update the market information database based on the updated data catalog. For example, embodiments of this application can collect market data from the new data catalog and update the data resource map based on the updated data catalog. Embodiments of this application can update the data flow map based on the updated data resource map and the market data from the new data catalog.

[0091] This application embodiment can improve and update the fine-tuning dataset based on the updated market information database. This application embodiment can use the updated fine-tuning dataset to update the metadata baseline model. In this way, this application embodiment can update the market information database in real time according to the update status of the data catalog, and improve the fine-tuning dataset in real time based on the updated market information database, thereby improving the accuracy of the metadata baseline model through incremental fine-tuning.

[0092] In this embodiment, at least one new metadata can be input into a metadata benchmark model, and feature scoring can be performed on the at least one new metadata based on the metadata benchmark model to obtain a score corresponding to each of the at least one new metadata. For example, in this embodiment, at least one new metadata can be input into an updated metadata benchmark model, and feature scoring can be performed on the at least one new metadata based on the updated metadata benchmark model to obtain a score corresponding to each of the at least one new metadata.

[0093] This application embodiment can update the metadata standard map based on at least one new metadata and its corresponding score. Thus, this application embodiment can update the metadata standard map when data catalogs are updated, thereby improving the metadata standard and enhancing its applicability.

[0094] As an example, embodiments of this application can perform semantic aggregation and classification on at least one new metadata and target metadata in a metadata standard graph to determine the target metadata corresponding to each new metadata. The target metadata corresponding to the new metadata has the same meaning as the new metadata.

[0095] The embodiments of this application can determine whether the score of each new metadata is higher than the score of the target metadata corresponding to each new metadata.

[0096] When the score of new metadata is higher than the score of the target metadata corresponding to the new metadata, embodiments of this application can use the new metadata as the metadata standard and update the metadata standard map. For example, embodiments of this application can directly establish a mapping relationship between the new metadata and the target metadata corresponding to the new metadata in the metadata standard map.

[0097] When the score of the new metadata is higher than the score of the target metadata corresponding to the new metadata, the embodiments of this application can establish a mapping relationship between the target metadata and the new metadata in the metadata standard map.

[0098] For example, such as Figure 3 As shown, the target metadata in the metadata standard graph is YN, and the other metadata are AN and VN. The new metadata is CN, which has the same meaning as YN, AN, and VN. When the score of CN is higher than that of YN, this embodiment of the application can use CN as the metadata standard, adding a mapping relationship between CN and YN to the metadata standard graph. For example, as... Figure 4 As shown, when the score of CN is lower than or equal to that of YN, this embodiment of the application can add a mapping relationship between CN and YN in the metadata standard map.

[0099] Based on the above embodiments, this application embodiment can extract features from each data directory in the data directory library to obtain the metadata of each data directory in the data directory library. Then, it scores the metadata extracted from the data directory library based on a metadata benchmark model, and selects the more universally applicable metadata from multiple metadata sources as the metadata standard between the various data directories in the data directory library according to the scores. In this way, this application embodiment can obtain a metadata standard that is compatible with multiple data directories in the data directory library, no longer limiting the number of participating data directories, reducing the difficulty of data exchange, thereby improving the universality of the metadata standard, and ultimately enabling standardized description and exchange of data, improving data quality and usability.

[0100] like Figure 5 As shown in the figure, this application provides a complete flowchart of a metadata processing method based on a large model, including the following steps:

[0101] S501: Retrieve directory content data from the data directory library.

[0102] S502: Extract structured features from the directory content data to obtain at least one metadata.

[0103] The process of determining at least one metadata in S502 is the same as the process of determining at least one metadata in step S102 above, and will not be repeated here.

[0104] S503: Perform semantic analysis on the target content data to obtain semantic analysis results corresponding to at least one metadata.

[0105] S504: Construct a metadata semantic graph based on at least one metadata and semantic analysis result.

[0106] S505: Construct a fine-tuning dataset based on the various data directories in the data catalog and the market data corresponding to each data directory.

[0107] There is no specific order of execution between S505 and S502.

[0108] In some embodiments, this application can construct a data resource map based on the correlation between various data directories in the data catalog library. The construction method of the data resource map is the same as that in the above embodiments and will not be repeated here. This application can expand the data resource map based on the market data corresponding to each data directory to obtain a data flow map. This application can construct a fine-tuning dataset based on the data flow map and the market data corresponding to each data directory.

[0109] S506: Update the metadata baseline model based on the fine-tuned dataset.

[0110] S507: Input the metadata semantic graph into the updated metadata benchmark model, and score at least one metadata feature based on the updated metadata benchmark model to obtain a score for at least one metadata.

[0111] S508: Perform semantic aggregation and classification on the metadata in the metadata semantic graph to obtain at least one metadata set.

[0112] S509: Determine the target metadata based on the score of each metadata in at least one metadata set.

[0113] In some embodiments, the metadata with the highest score in each metadata set can be used as the target metadata for each metadata set.

[0114] S510: Update the metadata standard graph based on the target metadata and other metadata in each metadata set.

[0115] In some embodiments, when metadata processing is performed on the data catalog in the data catalog for the first time, the metadata standard graph is empty. Embodiments of this application can directly construct the mapping relationship between the target metadata and other metadata to obtain the metadata standard graph.

[0116] In other embodiments, when metadata processing is not performed on the data catalog in the data catalog library for the first time, multiple metadata entries exist in the metadata standard graph. Embodiments of this application can determine whether the target metadata is standard metadata in the metadata standard graph. If yes, embodiments of this application can directly construct a mapping relationship between the target metadata and other metadata in the metadata standard graph. If no, embodiments of this application can use the target metadata as new standard metadata and construct a mapping relationship between the target metadata and standard metadata in the metadata graph, thus completing the update of the metadata standard graph.

[0117] S511: Determine whether the data catalog has been updated; if yes, proceed to step S501; if no, continue to step S511.

[0118] based on Figure 5 As shown in the embodiments of this application, the metadata of each data directory in the data catalog library can be scored, and the metadata with the highest universality can be selected as the standard metadata. This results in a metadata standard that is compatible with multiple data directories in the data catalog library, eliminating restrictions on the number of participating data directories, reducing the difficulty of data exchange, and thus improving the universality of the metadata standard. This, in turn, enables standardized description and exchange of data, improving data quality and usability. Furthermore, before scoring the metadata using the metadata benchmark model, this embodiment of the application optimizes and updates the metadata benchmark model using a fine-tuning dataset, thereby improving the scoring accuracy of the metadata benchmark model and ensuring the universality of the obtained metadata standard.

[0119] Based on the same inventive concept, this application also provides a metadata processing device based on a large model. Since the principle of this device in solving the problem is similar to the metadata processing method based on a large model in this application, the implementation of this device can refer to the implementation of the method, and the repeated parts will not be described again.

[0120] like Figure 6 As shown, this application provides a metadata processing apparatus based on a large model, including:

[0121] Acquisition unit 601 is used to acquire directory content data in the data directory library;

[0122] Extraction unit 602 is used to extract features from the directory content data to obtain data structure features, and extract at least one metadata from the directory content data based on the data structure features;

[0123] Scoring unit 603 is used to input the at least one metadata into a metadata benchmark model, perform feature scoring on the at least one metadata based on the metadata benchmark model, and obtain the score corresponding to each of the at least one metadata; the metadata benchmark model is a large model that has been fine-tuned and trained;

[0124] The determining unit 604 is used to determine target metadata based on the scores corresponding to each of the at least one metadata, wherein the target metadata is the metadata standard among the at least one metadata.

[0125] In one alternative implementation, before inputting the at least one metadata into the metadata benchmark model, the scoring unit 603 is further configured to:

[0126] Obtain a fine-tuning dataset, which includes a data flow graph and market data corresponding to each data directory in the data directory library; the market data corresponding to a data directory includes the evaluation information, usage information, and pricing information of that data directory;

[0127] The metadata baseline model is updated based on the fine-tuning algorithm and the fine-tuning dataset.

[0128] In one optional implementation, the scoring unit 603 is specifically used for:

[0129] Feature extraction is performed on the directory content data to obtain an embedding vector;

[0130] A data resource map is constructed based on the correlation between the embedded vectors; the data resource map is used to characterize the correlation between various data directories in the data catalog library;

[0131] Obtain the market data corresponding to each data directory, and obtain the data flow map based on the market data and the data resource map;

[0132] The fine-tuning dataset is constructed based on the data flow map and the market data.

[0133] In one optional implementation, the determining unit 604 is specifically used for:

[0134] Perform semantic aggregation and classification on at least one metadata in the directory content data to obtain at least one metadata set;

[0135] For any one of the at least one metadata set, the target metadata corresponding to the metadata set is determined based on the score corresponding to each metadata in the metadata set.

[0136] In one optional implementation, the determining unit 604 is specifically used for:

[0137] Semantic analysis is performed on the data related to the at least one metadata in the directory content data to obtain the semantic analysis results corresponding to each of the at least one metadata.

[0138] The semantic analysis results are aggregated and categorized to obtain at least one metadata set.

[0139] In one optional implementation, the determining unit 604 is further configured to:

[0140] A metadata standard graph is constructed based on the target metadata and other metadata in each metadata set; the other metadata refers to the metadata in each metadata set other than the target metadata, and the metadata standard graph is used to characterize the mapping relationship between the target metadata and the other metadata in each metadata set.

[0141] In one optional implementation, the acquisition unit 601 is further configured to: determine the new directory content data of the new data directory in the data directory when the data directory is updated;

[0142] The extraction unit 602 is also used to: extract features from the new directory content data to obtain at least one new metadata;

[0143] The scoring unit 603 is also used to: input the at least one new metadata into the metadata benchmark model, perform feature scoring on the at least one new metadata based on the metadata benchmark model, and obtain the scores corresponding to each of the at least one new metadata.

[0144] The determining unit 604 is further configured to: update the metadata standard map based on the at least one new metadata and the scores corresponding to each of the at least one new metadata.

[0145] An optional implementation method is to train the metadata baseline model in the following way:

[0146] Obtain a sample dataset, which includes multiple standard metadata;

[0147] The large model is transferred to the sample dataset to obtain the metadata baseline model.

[0148] and Figure 1The metadata processing method based on a large model shown is based on the same inventive concept, and embodiments of this application provide an electronic device. For example... Figure 7 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method embodiment section of this application. The electronic device can be a server or a terminal device.

[0149] The electronic device includes at least a memory for storing data and a processor for data processing. The processor for data processing can be implemented using a microprocessor, CPU, GPU (Graphics Processing Unit), DSP, or FPGA. The memory stores operation instructions, which can be computer-executable code, to implement the various steps in the metadata processing method based on a large model according to the embodiments of this application.

[0150] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; as shown below. Figure 7 As shown, in this embodiment of the application, the electronic device 100 includes: a processor 101, a display 102, a memory 103, an input device 106, a bus 105, and a communication module 104; the processor 101, memory 103, input device 106, display 102, and communication module 104 are all connected through the bus 105, which is used for data transmission between the processor 101, memory 103, display 102, communication module 104, and input device 106.

[0151] The memory 103 can be used to store software programs and modules, such as the program instructions / modules corresponding to the metadata processing method based on a large model in this embodiment. The processor 101 executes various functional applications and data processing of the electronic device 100 by running the software programs and modules stored in the memory 103, such as the metadata processing method based on a large model provided in this embodiment. The memory 103 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs of at least one application, etc.; the data storage area may store data created according to the use of the electronic device 100 (such as data catalogs, metadata standard maps, market data, data flow maps, and other related data). In addition, the memory 103 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0152] The processor 101 is the control center of the electronic device 100. It connects various parts of the electronic device 100 via the bus 105 and various interfaces and lines. It performs various functions of the electronic device 100 and processes data by running or executing software programs and / or modules stored in the memory 103 and calling data stored in the memory 103. Optionally, the processor 101 may include one or more processing units, such as a CPU, GPU, digital processing unit, etc.

[0153] The processor 101 can display the metadata standard map to the user via the display 102.

[0154] The processor 101 can also connect to the network via the communication module 104 to obtain market data, etc.

[0155] Input device 106 is mainly used to obtain user input operations, and the input device 106 may also be different depending on the electronic device. For example, when the electronic device is a computer, the input device 106 can be an input device such as a mouse or keyboard; when the electronic device is a smartphone, tablet computer, or other portable device, the input device 106 can be a touch screen.

[0156] In some possible implementations, various aspects of the large-model-based metadata processing method provided in this application can also be implemented as a program product comprising program code. When the program product is run on a computer device, the program code causes the computer device to perform the steps of the large-model-based metadata processing method according to the various exemplary embodiments of this application described above. For example, the computer device can perform actions such as... Figure 1 The flowchart of the metadata processing method based on the large model is shown in steps S101 to S104.

[0157] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0158] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A metadata processing method based on a large model, characterized in that, include: Retrieve directory content data from the data directory library; Feature extraction is performed on the directory content data to obtain data structure features, and at least one metadata is extracted from the directory content data based on the data structure features; The at least one metadata is input into the metadata benchmark model, and feature scoring is performed on the at least one metadata based on the metadata benchmark model to obtain the score corresponding to each of the at least one metadata. The metadata baseline model is a large model that has been fine-tuned and trained; Based on the scores corresponding to each of the at least one metadata, target metadata is determined, wherein the target metadata is the metadata standard among the at least one metadata.

2. The method according to claim 1, characterized in that, Before inputting the at least one metadata into the metadata baseline model, the method further includes: Obtain a fine-tuning dataset, which includes a data flow graph and market data corresponding to each data directory in the data directory library; the market data corresponding to a data directory includes the evaluation information, usage information, and pricing information of that data directory; The metadata baseline model is updated based on the fine-tuning algorithm and the fine-tuning dataset.

3. The method according to claim 2, characterized in that, The process of obtaining the fine-tuning dataset includes: Feature extraction is performed on the directory content data to obtain an embedding vector; A data resource map is constructed based on the correlation between the embedded vectors; the data resource map is used to characterize the correlation between various data directories in the data catalog library; Obtain the market data corresponding to each data directory, and obtain the data flow map based on the market data and the data resource map; The fine-tuning dataset is constructed based on the data flow map and the market data.

4. The method according to claim 3, characterized in that, The step of determining the target metadata based on the scores corresponding to each of the at least one metadata includes: Perform semantic aggregation and classification on at least one metadata in the directory content data to obtain at least one metadata set; For any one of the at least one metadata set, the target metadata corresponding to the metadata set is determined based on the score corresponding to each metadata in the metadata set.

5. The method according to claim 4, characterized in that, The semantic aggregation and classification of at least one metadata in the directory content data yields at least one metadata set, including: Semantic analysis is performed on the data related to the at least one metadata in the directory content data to obtain the semantic analysis results corresponding to each of the at least one metadata. The semantic analysis results are aggregated and categorized to obtain at least one metadata set.

6. The method according to claim 5, characterized in that, The method further includes: A metadata standard graph is constructed based on the target metadata and other metadata in each metadata set; the other metadata refers to the metadata in each metadata set other than the target metadata, and the metadata standard graph is used to characterize the mapping relationship between the target metadata and the other metadata in each metadata set.

7. The method according to claim 6, characterized in that, The method further includes: When it is determined that the data directory library is updated, the new directory content data of the new data directory in the data directory library is determined; Feature extraction is performed on the new directory content data to obtain at least one new metadata. The at least one new metadata is input into the metadata benchmark model, and the at least one new metadata is scored based on the metadata benchmark model to obtain the score corresponding to each of the at least one new metadata. The metadata standard map is updated based on the at least one new metadata and the corresponding score of each of the at least one new metadata.

8. The method according to any one of claims 1-7, characterized in that, The metadata baseline model is trained using the following methods: Obtain a sample dataset, which includes multiple standard metadata; The large model is transferred to the sample dataset to obtain the metadata baseline model.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, it implements the method of any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 8.

Images