Metadata processing method and apparatus, server, and storage medium

Abstract

The application provides a metadata processing method and device, a server and a storage medium. The method comprises the following steps: receiving a plurality of metadata sent by a data terminal; splicing the field name and field description information of each metadata to obtain spliced data, and determining the spliced data as sample metadata to obtain a plurality of sample metadata; segmenting each sample metadata to obtain a plurality of corresponding segmented characters and saving the segmented characters to a dictionary table; obtaining a plurality of index values of the plurality of segmented characters in the dictionary table; inputting the plurality of segmented characters and the plurality of index values into an embedding layer, passing through an editor layer and a full connection layer to output the probability distribution data of each sample metadata in a plurality of different categories; calculating the loss value of the probability distribution data and the real category distribution data of each sample metadata, and optimizing the initial metadata classification model according to the loss value to obtain a final metadata classification model. The method improves the accuracy of metadata classification.

Description

Technical Field

[0001] This application relates to the field of big data processing, and in particular to a metadata processing method, apparatus, server, and storage medium. Background Technology

[0002] With the development of the internet, enterprises and organizations generate massive amounts of data every day. How to manage this data, and the governance of metadata, has become a focus of attention. The classification of metadata is the foundation of metadata management, data governance, and data security.

[0003] Currently, existing technologies mainly involve extracting features from the metadata dataset to construct a metadata classification model in order to classify metadata.

[0004] However, this metadata classification model reduces the accuracy of metadata classification. Summary of the Invention

[0005] This application provides a metadata processing method, apparatus, server, and storage medium to solve the technical problem of low metadata classification accuracy.

[0006] Firstly, this application provides a metadata processing method, including:

[0007] Receive multiple metadata packets sent by the data terminal, where each metadata packet includes a field name and a field description.

[0008] The field name and field description information of each metadata are concatenated to obtain concatenated data, and the concatenated data is determined as sample metadata to obtain multiple sample metadata.

[0009] Each sample metadata is segmented to obtain multiple segmentation characters corresponding to each sample metadata, and these multiple segmentation characters are saved to a dictionary table.

[0010] Obtain multiple index values ​​of multiple delimiters in the dictionary table, where each delimiter corresponds to one index value.

[0011] The multiple delimiters and multiple index values ​​are input into the embedding layer to output the embedding layer result of each sample metadata.

[0012] The embedding layer results are input to the editor layer to output the editor layer results of each sample metadata.

[0013] The editor results are input into a fully connected layer to output the probability distribution data of each sample's metadata across multiple different categories.

[0014] Calculate the loss value of the probability distribution data and the true category distribution data of each sample metadata, and optimize the initial metadata classification model based on the loss value to obtain the final metadata classification model; wherein the final metadata classification model is used to classify the metadata to be classified.

[0015] Optionally, in the method described above, the step of segmenting each sample metadata to obtain multiple segmentation characters corresponding to each sample metadata includes: segmenting each sample metadata according to a preset character to obtain multiple initial segmentation characters; determining all numeric characters in the multiple initial segmentation characters as characters to be replaced; and replacing the characters to be replaced with a preset replacement character to obtain multiple segmentation characters corresponding to each sample metadata.

[0016] Optionally, in the method described above, the embedding layer includes a Token embedding layer and a Position embedding layer; correspondingly, the step of inputting the plurality of delimiters and the plurality of index values ​​into the embedding layer to output the embedding layer result of each sample metadata includes: inputting the plurality of delimiters and the plurality of index values ​​into the Token embedding layer of the embedding layer, such that the Token embedding layer converts each sample metadata into data of the Token sequence type according to the plurality of delimiters and the plurality of index values; determining whether it is necessary to input the plurality of delimiters and the plurality of index values ​​into the Position embedding layer of the embedding layer; if it is determined that it is necessary, then inputting the plurality of delimiters and the plurality of index values ​​into the Position embedding layer of the embedding layer, such that the Position embedding layer converts each sample metadata into data of the Position sequence type according to the plurality of delimiters and the plurality of index values; and determining the embedding layer result of each sample metadata according to the Token sequence type data and the Position sequence type data.

[0017] Optionally, in the method described above, the editor layer includes: multiple convolutional layers and a global multi-head attention layer; correspondingly, the step of inputting the embedding layer result to the editor layer to output the editor layer result of each sample metadata includes: inputting the embedding layer result to the multiple convolutional layers of the editor layer to output multiple convolutional layer results; concatenating the multiple convolutional layer results to obtain the final convolutional layer result; inputting the convolutional layer result to the global multi-head attention layer of the editor layer to output the global multi-head attention layer result; and determining the global multi-head attention layer result as the editor layer result of each sample metadata.

[0018] Optionally, in the method described above, calculating the loss value of the probability distribution data and the true category distribution data of each sample metadata, and optimizing the initial metadata classification model based on the loss value to obtain the final metadata classification model, includes: using a loss function to calculate the loss value of the probability distribution data and the true category distribution data of each sample metadata; iteratively optimizing the initial metadata classification model based on the loss value to obtain an optimized metadata classification model; and determining the optimized metadata classification model as the final metadata classification model.

[0019] Optionally, in the method described above, the step of iteratively optimizing the initial metadata classification model based on the loss value to obtain an optimized metadata classification model includes: calculating the gradient using an optimization algorithm to obtain parameters for updating the initial metadata classification model; updating the parameters and calculating the updated loss value; if it is determined that the loss value before the update and the loss value after the update are the same, then the iteration ends to obtain the optimized metadata classification model.

[0020] Optionally, the method described above further includes: inputting the metadata to be classified into the final metadata classification model to output the probability distribution data of the metadata to be classified; and determining the category corresponding to the maximum value in the probability distribution data as the category of the metadata to be classified.

[0021] Secondly, this application provides a metadata processing apparatus, comprising:

[0022] The receiving module is used to receive multiple metadata sent by the data terminal, where each metadata includes a field name and a field description.

[0023] The first acquisition module is used to concatenate the field name and field description information of each metadata to obtain concatenated data, and to determine the concatenated data as sample metadata to obtain multiple sample metadata.

[0024] The second acquisition module is used to segment each sample metadata to obtain multiple segmentation characters corresponding to each sample metadata, and save the multiple segmentation characters to a dictionary table.

[0025] The third acquisition module is used to acquire multiple index values ​​of multiple delimiters in the dictionary table, wherein each delimiter corresponds to one index value.

[0026] The first output module is used to input the plurality of delimiters and the plurality of index values ​​into the embedding layer to output the embedding layer result of each sample metadata.

[0027] The second output module is used to input the embedding layer results to the editor layer to output the editor layer results of each sample metadata.

[0028] The third output module is used to input the editor results into the fully connected layer to output the probability distribution data of each sample metadata in multiple different categories.

[0029] The fourth acquisition module is used to calculate the loss value of the probability distribution data and the true category distribution data of each sample metadata, and optimize the initial metadata classification model according to the loss value to obtain the final metadata classification model; wherein the final metadata classification model is used to classify the metadata to be classified.

[0030] Thirdly, this application provides a server, comprising:

[0031] At least one processor and memory;

[0032] The memory stores computer-executed instructions;

[0033] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the metadata processing method as described in the first aspect and various possible designs of the first aspect.

[0034] Fourthly, this application provides a computer storage medium storing computer execution instructions, which, when executed by a processor, implement the metadata processing method described in the first aspect and various possible designs of the first aspect.

[0035] The metadata processing method, apparatus, server, and storage medium provided in this application concatenate the field names and field descriptions in each piece of metadata, and determine the concatenated data as sample metadata; the sample metadata is then segmented into multiple delimiters and saved to a dictionary table; the multiple delimiters and their index values ​​in the dictionary table are processed through an embedding layer, an editor layer, and a fully connected layer to obtain the probability distribution data of each sample metadata in multiple different categories; the loss value between the sample metadata and the actual category distribution data is calculated, and the initial metadata classification model is optimized based on the loss value to obtain the final metadata classification model. This avoids the impact of irregular and inconsistent metadata naming on the accuracy of processing field names and field descriptions, improves the accuracy of the metadata classification model, and thus improves the accuracy of metadata classification. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0037] Figure 1 This is a schematic diagram of a metadata processing system provided in an embodiment of this application.

[0038] Figure 2 A flowchart illustrating a metadata processing method provided in one embodiment of this application;

[0039] Figure 3 This is a schematic diagram of the metadata processing apparatus provided in the embodiments of this application;

[0040] Figure 4 This is a schematic diagram of the hardware structure of the server provided in an embodiment of this application.

[0041] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0042] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0043] With the development of the internet, enterprises and organizations generate massive amounts of data every day. How to manage this data, and the governance of metadata, has become a focus of attention. Metadata classification is the foundation of metadata management, data governance, and data security. Currently, existing technologies mainly rely on extracting features from metadata datasets to construct metadata classification models. However, this type of metadata classification model reduces the accuracy of metadata classification.

[0044] To address the aforementioned technical problems, this application proposes the following technical approach: Considering that constructing a metadata classification model by extracting features from the metadata dataset can reduce the accuracy of metadata classification due to irregular and inconsistent metadata naming, the inventors devised a method to concatenate and segment field names and field descriptions, save multiple segmentation characters to a dictionary table, and combine an embedding layer, an editor layer, and a fully connected layer to construct a metadata classification model, thereby improving the accuracy of metadata classification.

[0045] The metadata processing method provided in this application aims to solve the above-mentioned technical problems in the prior art.

[0046] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0047] Figure 1 This is a schematic diagram illustrating an application scenario for metadata processing provided in an embodiment of this application. For example... Figure 1 As shown, the application scenario includes: data terminal 101, server 102 and display terminal 103.

[0048] The data terminal 101 can be a computer or a terminal device.

[0049] Server 102 can be a single server or a cluster of multiple servers.

[0050] Platform device 103 can display terminals, etc.

[0051] refer to Figure 1 Data terminal 101 sends multiple metadata to server 102; server 102 performs a series of processes on the multiple metadata to obtain multiple sample data; after passing through the embedding layer, editor layer and fully connected layer, it obtains the probability distribution data of each sample metadata in multiple different categories; and calculates the loss value with the real category distribution data, and optimizes the initial metadata classification model according to the loss value to obtain the final metadata classification model; the final metadata classification model is used to classify the metadata to be classified, and sends the metadata category to display terminal 103 for display.

[0052] Figure 2 This is a flowchart illustrating a metadata processing method provided in one embodiment of this application. The execution entity of this embodiment can be... Figure 1 The server 102 in this embodiment can also be other servers with similar functions; this embodiment does not impose any special limitations here. Figure 2 As shown, the method includes:

[0053] S201: Receive multiple metadata sent by the data terminal, wherein each metadata includes a field name and a field description.

[0054] In this embodiment, metadata refers to information in the database such as database name, database description, table name, table description, field name, field description, field type, and maximum field length. These metadata elements generally have the following characteristics: a. They contain both Chinese and English text: field names are typically named using English words or at least one abbreviation or shortened form, while field descriptions are generally in Chinese; b. They are short in length: field names are generally 3 to 5 words, and field descriptions are generally sentences of less than 20 characters; c. The naming conventions are irregular and inconsistent.

[0055] For example, the field name is "local_calling_dura_n12" and the field description is "local incoming call duration in the past 12 months".

[0056] S202: Concatenate the field name and field description information of each metadata to obtain concatenated data, and determine the concatenated data as sample metadata to obtain multiple sample metadata.

[0057] Specifically, the field name and field description are concatenated with a space.

[0058] S203: Segment each sample metadata to obtain multiple delimiter characters corresponding to each sample metadata, and save the multiple delimiter characters to the dictionary table.

[0059] In this embodiment, vocab.txt from the Bert Chinese model is used as the dictionary table.

[0060] Specifically, step S203 includes S2031 to S2032:

[0061] S2031: Segment each sample metadata according to preset characters to obtain multiple initial segmentation characters.

[0062] Specifically, the field names in each sample metadata are separated by underscores and numbers, and the field descriptions in each sample metadata are separated by characters.

[0063] For example, the field names in the sample metadata “local_calling_dura_n12 local called call duration in the past 12 months” are separated by “_” and “\d+”, and the field description information is separated by character. The multiple initial separator characters after separation are “locall calling dura n 12 local called call duration in the past 12 months”.

[0064] S2032: Identify all numeric characters among the multiple initial delimiters as the characters to be replaced.

[0065] For example, “12”, “1” and “2” are identified as the characters to be replaced.

[0066] S2033: Replace the character to be replaced with the preset replacement character to obtain multiple delimiter characters corresponding to the metadata of each sample.

[0067] Specifically, the default replacement character is [dight].

[0068] In this embodiment, multiple delimiters are saved to the dictionary table, and each delimiter is added at most once. "[digit]" is also saved to the dictionary table.

[0069] S204: Retrieve multiple index values ​​of multiple delimiters in the dictionary table, where each delimiter corresponds to one index value.

[0070] S205: Input multiple delimiters and multiple index values ​​into the embedding layer to output the embedding layer result of each sample's metadata.

[0071] The embedding layer includes a Token embedding layer and a Position embedding layer.

[0072] Specifically, step S205 includes S2051 to S2054:

[0073] S2051: Input multiple delimiters and multiple index values ​​into the Token embedding layer of the embedding layer, so that the Token embedding layer converts the metadata of each sample into data of the Token sequence type according to the multiple delimiters and multiple index values.

[0074] In the Token embedding layer, each delimiter character is converted into a vector of fixed dimensions.

[0075] Specifically, if the length of multiple delimiters is less than the dimension of the embedding layer, they are padded with "[PAD]". If the length of multiple delimiters is greater than the dimension of the embedding layer, the delimiters of the first dimension are selected. Based on the index value of each delimiter, the data is processed through a network of dictionary length × dimension size to obtain the token sequence type data.

[0076] S2052: Determine whether multiple delimiter characters and multiple index values ​​need to be input into the Position embedding layer of the embedding layer.

[0077] S2053: If necessary, input multiple delimiter characters and multiple index values ​​into the Position embedding layer of the embedding layer, so that the Position embedding layer converts the metadata of each sample into data of the Position sequence type according to the multiple delimiter characters and multiple index values.

[0078] The Position embedding layer addresses the positional encoding problem by attaching a different vector to the segmentation characters at different positions for differentiation. The Position embedding layer is calculated using the Position embedding layer calculation method of the Transformer model.

[0079] Optionally, the Position embedding layer is optional.

[0080] S2054: Determine the embedding layer result of each sample's metadata based on the data of the Token sequence type and the data of the Position sequence type.

[0081] Specifically, the data of the Token sequence type and the data of the Position sequence type are added together and then normalized by the layer to serve as the embedding layer result of the metadata of each sample.

[0082] In this embodiment, if it is determined that multiple delimiters and multiple index values ​​do not need to be input into the Position embedding layer of the embedding layer, then the data of the Token sequence type is used as the embedding layer result.

[0083] S206: Input the embedding layer results into the editor layer to output the editor layer results for each sample's metadata.

[0084] The editor layer includes multiple convolutional layers and a global multi-head attention layer.

[0085] Specifically, step S206 includes S2061 to S2064:

[0086] S2061: Input the embedding layer results into multiple convolutional layers of the editor layer to output multiple convolutional layer results.

[0087] Each convolutional layer contains a different convolutional kernel, consisting of multiple convolutional kernels, an activation function, and one pooling layer. The default number of convolutional layers is three, and the number of global attention heads is twice the number of convolutional kernels by default; optionally, both of these parameters can be used as hyperparameters for debugging the model.

[0088] Specifically, the embedding layer results are fed into different convolutional layers. In each convolutional layer, convolution operations are performed using corresponding convolutional kernels. An activation function is used to perform non-linear activation on the results of the convolution operations. The output after convolution and activation operations is then reduced in dimensionality by a pooling layer.

[0089] Alternatively, the activation function can be ReLU.

[0090] S2062: Concatenate the results of multiple convolutional layers to obtain the final convolutional layer result.

[0091] Specifically, the outputs of all convolutional layers are pooled and then concatenated together to form the final convolutional layer result.

[0092] S2063: Input the convolutional layer results into the global multi-head attention layer of the editor layer to output the global multi-head attention layer results.

[0093] S2064: Determine the global multi-head attention layer results as the editor layer results for each sample's metadata.

[0094] S207: Input the editor results into a fully connected layer to output the probability distribution data of each sample's metadata across multiple different categories.

[0095] The fully connected layer acts as a classifier, and its output dimension is the number of categories of multiple sample metadata, thus obtaining the probability distribution data of each sample metadata in multiple different categories.

[0096] S208: Calculate the loss value of the probability distribution data and the true category distribution data of each sample's metadata, and optimize the initial metadata classification model based on the loss value to obtain the final metadata classification model; the final metadata classification model is used to classify the metadata to be classified.

[0097] Optionally, the model parameters can be updated by minimizing the loss value to optimize the initial metadata classification model and obtain the final metadata classification model.

[0098] In summary, this embodiment concatenates the field names and descriptions of each metadata element and identifies the concatenated data as sample metadata. The sample metadata is then segmented into multiple delimiters and stored in a dictionary table. These delimiters, along with their index values ​​in the dictionary table, are processed through an embedding layer, an editor layer, and a fully connected layer to obtain the probability distribution data for each sample metadata element across multiple categories. The loss value between this distribution and the actual category distribution data is calculated, and the initial metadata classification model is optimized based on this loss value to obtain the final metadata classification model. This approach avoids the inaccuracy of processing field names and descriptions caused by irregular and inconsistent metadata naming, improving the accuracy of the metadata classification model and thus increasing the overall accuracy of metadata classification.

[0099] In another embodiment of this application, based on the above embodiments, the process of optimizing the initial metadata classification model according to the loss value to obtain the final metadata classification model is described in detail. The execution subject of this embodiment can be... Figure 1 The server in the illustrated embodiment can also be other servers with similar functions. This embodiment does not impose any special limitations here, and specifically includes:

[0100] Sa: The loss function is used to calculate the loss value of the probability distribution data and the true class distribution data of each sample's metadata.

[0101] The loss function measures the difference between the probability distribution data and the true class distribution data; the loss value is the specific numerical value of the calculated loss function.

[0102] Optionally, the loss function can be the cross-entropy loss function.

[0103] Sb: Based on the loss value, the initial metadata classification model is iteratively optimized to obtain an optimized metadata classification model.

[0104] Specifically, an optimization algorithm is used to calculate the gradient and obtain the parameters for updating the initial metadata classification model; the parameters are updated and the updated loss value is calculated; if the loss value before and after the update are the same, the iteration ends to obtain the optimized metadata classification model.

[0105] By minimizing the loss value, the model can be made closer to the true class distribution, thereby improving the accuracy of the metadata classification model. In each iteration, the backpropagation algorithm is used to update the model to reduce the loss value.

[0106] Backpropagation is an optimization algorithm used to train neural networks. It updates parameters and optimizes the model by calculating the gradient of the loss function with respect to each parameter in the network.

[0107] Optionally, the iteration can also end when the maximum number of iterations is reached.

[0108] Sc: The optimized metadata classification model is determined as the final metadata classification model.

[0109] In this embodiment, the metadata to be classified is input into the final metadata classification model to output the probability distribution data of the metadata to be classified; the category corresponding to the maximum value in the probability distribution data is determined as the category of the metadata to be classified.

[0110] Specifically, mathematical functions are used to obtain the index value corresponding to the maximum value in the probability distribution data, and the category of the metadata to be classified is obtained according to the category dictionary.

[0111] Alternatively, the mathematical function can be the argmax function.

[0112] In summary, this embodiment improves the accuracy of the metadata classification model by iteratively optimizing the initial metadata classification model, thereby increasing the accuracy of metadata classification.

[0113] Figure 3This is a schematic diagram of the metadata processing apparatus provided in an embodiment of this application. Figure 3 As shown, the metadata processing device includes: a receiving module 301, a first acquisition module 302, a second acquisition module 303, a third acquisition module 304, a first output module 305, a second output module 306, a third output module 307, and a fourth acquisition module 308.

[0114] The receiving module 301 is used to receive multiple metadata sent by the data terminal, wherein each metadata includes a field name and a field description.

[0115] The first acquisition module 302 is used to concatenate the field name and field description information of each metadata to obtain concatenated data, and to determine the concatenated data as sample metadata in order to obtain multiple sample metadata.

[0116] The second acquisition module 303 is used to segment the metadata of each sample, obtain multiple segmentation characters corresponding to the metadata of each sample, and save the multiple segmentation characters to the dictionary table.

[0117] The third acquisition module 304 is used to acquire multiple index values ​​of multiple delimiters in the dictionary table, wherein each delimiter corresponds to one index value.

[0118] The first output module 305 is used to input the plurality of segmentation characters and the plurality of index values ​​into the embedding layer to output the embedding layer result of each sample metadata.

[0119] The second output module 306 is used to input the embedding layer result to the editor layer to output the editor layer result of each sample metadata.

[0120] The third output module 307 is used to input the editor result into the fully connected layer to output the probability distribution data of each sample metadata in multiple different categories.

[0121] The fourth acquisition module 308 is used to calculate the loss value of the probability distribution data and the true category distribution data of each sample metadata, and optimize the initial metadata classification model according to the loss value to obtain the final metadata classification model; wherein the final metadata classification model is used to classify the metadata to be classified.

[0122] Optionally, in the method described above, the second acquisition module 303 is specifically used to: segment each sample metadata according to a preset character to obtain multiple initial segmentation characters; determine all numeric characters in the multiple initial segmentation characters as characters to be replaced; replace the characters to be replaced with a preset replacement character to obtain multiple segmentation characters corresponding to each sample metadata.

[0123] Optionally, in the method described above, the embedding layer includes a Token embedding layer and a Position embedding layer; correspondingly, the first output module 305 is specifically configured to: input the plurality of delimiters and the plurality of index values ​​into the Token embedding layer of the embedding layer, such that the Token embedding layer converts each sample metadata into Token sequence type data according to the plurality of delimiters and the plurality of index values; determine whether it is necessary to input the plurality of delimiters and the plurality of index values ​​into the Position embedding layer of the embedding layer; if it is determined that it is necessary, input the plurality of delimiters and the plurality of index values ​​into the Position embedding layer of the embedding layer, such that the Position embedding layer converts each sample metadata into Position sequence type data according to the plurality of delimiters and the plurality of index values; and determine the embedding layer result of each sample metadata according to the Token sequence type data and the Position sequence type data.

[0124] Optionally, in the method described above, the editor layer includes: multiple convolutional layers and a global multi-head attention layer; correspondingly, the second output module 306 is specifically used for: inputting the embedding layer result to the multiple convolutional layers of the editor layer to output multiple convolutional layer results; concatenating the multiple convolutional layer results to obtain the final convolutional layer result; inputting the convolutional layer result to the global multi-head attention layer of the editor layer to output the global multi-head attention layer result; and determining the global multi-head attention layer result as the editor layer result of each sample metadata.

[0125] Optionally, in the method described above, the fourth acquisition module 308 is specifically used to: calculate the loss value of the probability distribution data and the true category distribution data of each sample metadata using a loss function; iteratively optimize the initial metadata classification model based on the loss value to obtain an optimized metadata classification model; and determine the optimized metadata classification model as the final metadata classification model.

[0126] Optionally, in the method described above, the fourth acquisition module 308 is further configured to: calculate the gradient using an optimization algorithm to obtain parameters for updating the initial metadata classification model; update the parameters and calculate the updated loss value; if the loss value before the update and the loss value after the update are the same, then the iteration ends to obtain the optimized metadata classification model.

[0127] Optionally, in the method described above, the apparatus further includes a determining module 309, configured to: input the metadata to be classified into the final metadata classification model to output the probability distribution data of the metadata to be classified; and determine the category corresponding to the maximum value in the probability distribution data as the category of the metadata to be classified.

[0128] Figure 4 The hardware structure diagram of the server provided in this application embodiment is shown below. Figure 4 As shown, the server in this embodiment includes: at least one processor 401 and a memory 402; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the metadata processing method described above.

[0129] Alternatively, the memory 402 can be either standalone or integrated with the processor 401.

[0130] When the memory 402 is set up independently, the server also includes a bus 403 for connecting the memory 402 and the processor 401.

[0131] This application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the metadata processing method described above.

[0132] This application also provides a computer program product, including a computer program stored in a computer storage medium, at least one processor can read the computer program from the computer storage medium, and at least one processor can implement the above-mentioned metadata processing method when executing the computer program.

[0133] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0134] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0135] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0136] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0137] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0138] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0139] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0140] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0141] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A metadata processing method, characterized by, Applied to servers, including: Receive multiple metadata sent by the data terminal, where each metadata includes a field name and a field description; The field name and field description information of each metadata are concatenated to obtain concatenated data, and the concatenated data is determined as sample metadata to obtain multiple sample metadata. Each sample metadata is segmented to obtain multiple segmentation characters corresponding to each sample metadata, and the multiple segmentation characters are saved to a dictionary table; Obtain multiple index values ​​of multiple delimiters in the dictionary table, where each delimiter corresponds to one index value; The plurality of delimiters and the plurality of index values ​​are input into the embedding layer to output the embedding layer result of each sample metadata; The embedding layer results are input to the editor layer to output the editor layer results of each sample metadata; The results from the editor layer are input into the fully connected layer to output the probability distribution data of each sample's metadata across multiple different categories; Calculate the loss value between the probability distribution data and the true category distribution data of each sample's metadata, and optimize the initial metadata classification model based on the loss value to obtain the final metadata classification model; wherein the final metadata classification model is used to classify the metadata to be classified. The embedding layer includes: a Token embedding layer and a Position embedding layer; Accordingly, the step of inputting the plurality of segmentation characters and the plurality of index values ​​into the embedding layer to output the embedding layer result of each sample metadata includes: The plurality of delimiting characters and the plurality of index values ​​are input into the Token embedding layer of the embedding layer, so that the Token embedding layer converts the metadata of each sample into data of the Token sequence type according to the plurality of delimiting characters and the plurality of index values; Determine whether the multiple delimiters and multiple index values ​​need to be input into the Position embedding layer of the embedding layer; If necessary, the plurality of delimiting characters and the plurality of index values ​​are input to the Position embedding layer of the embedding layer, so that the Position embedding layer converts the metadata of each sample into data of the Position sequence type according to the plurality of delimiting characters and the plurality of index values; Based on the data of the Token sequence type and the data of the Position sequence type, determine the embedding layer result of the metadata of each sample; The editor layer includes: multiple convolutional layers and a global multi-head attention layer; Accordingly, the step of inputting the embedding layer result to the editor layer to output the editor layer result of each sample metadata includes: The embedding layer results are input to the multiple convolutional layers of the editor layer to output multiple convolutional layer results; The results of the multiple convolutional layers are concatenated to obtain the final convolutional layer result; The results of the convolutional layer are input into the global multi-head attention layer of the editor layer to output the results of the global multi-head attention layer; The results of the global multi-head attention layer are determined as the editor layer results of the metadata of each sample.

2. The method according to claim 1, characterized in that, The step of segmenting each sample metadata to obtain multiple segmentation characters corresponding to each sample metadata includes: Each sample metadata is segmented according to preset characters to obtain multiple initial segmentation characters; All numeric characters among the multiple initial dividing characters are identified as characters to be replaced; The preset replacement character is used to replace the character to be replaced, resulting in multiple delimiter characters corresponding to each sample metadata.

3. The method according to claim 1 or 2, characterized in that, The process of calculating the loss value of the probability distribution data and the true category distribution data of each sample's metadata, and optimizing the initial metadata classification model based on the loss value to obtain the final metadata classification model, includes: Using a loss function, calculate the loss value of the probability distribution data and the true category distribution data of each sample's metadata; Based on the loss value, the initial metadata classification model is iteratively optimized to obtain an optimized metadata classification model; The optimized metadata classification model is determined as the final metadata classification model.

4. The method according to claim 3, characterized in that, The step of iteratively optimizing the initial metadata classification model based on the loss value to obtain an optimized metadata classification model includes: The gradient is calculated using an optimization algorithm to obtain the parameters for updating the initial metadata classification model; The parameters are updated, and the updated loss value is calculated. If the loss value before the update and the loss value after the update are determined to be the same, the iteration ends to obtain an optimized metadata classification model.

5. The method according to claim 1, characterized in that, Also includes: The metadata to be classified is input into the final metadata classification model to output the probability distribution data of the metadata to be classified. The category corresponding to the maximum value in the probability distribution data is determined as the category of the metadata to be classified.

6. A metadata processing apparatus, characterized in that, Applied to servers, including: The receiving module is used to receive multiple metadata sent by the data terminal, wherein each metadata includes a field name and a field description. The first acquisition module is used to concatenate the field name and field description information of each metadata to obtain concatenated data, and to determine the concatenated data as sample metadata to obtain multiple sample metadata. The second acquisition module is used to segment the metadata of each sample to obtain multiple segmentation characters corresponding to each sample metadata, and save the multiple segmentation characters to a dictionary table; The third acquisition module is used to acquire multiple index values ​​of multiple delimiters in the dictionary table, wherein each delimiter corresponds to one index value; The first output module is used to input the plurality of delimiters and the plurality of index values ​​into the embedding layer to output the embedding layer result of each sample metadata; The second output module is used to input the embedding layer result to the editor layer to output the editor layer result of each sample metadata; The third output module is used to input the results of the editor layer into the fully connected layer to output the probability distribution data of each sample metadata in multiple different categories; The fourth acquisition module is used to calculate the loss value of the probability distribution data and the true category distribution data of each sample metadata, and to optimize the initial metadata classification model based on the loss value to obtain the final metadata classification model; wherein the final metadata classification model is used to classify the metadata to be classified. The embedding layer includes a Token embedding layer and a Position embedding layer. The first output module is specifically used for: The plurality of delimiting characters and the plurality of index values ​​are input into the Token embedding layer of the embedding layer, so that the Token embedding layer converts the metadata of each sample into data of the Token sequence type according to the plurality of delimiting characters and the plurality of index values; Determine whether the multiple delimiters and multiple index values ​​need to be input into the Position embedding layer of the embedding layer; If necessary, the plurality of delimiting characters and the plurality of index values ​​are input to the Position embedding layer of the embedding layer, so that the Position embedding layer converts the metadata of each sample into data of the Position sequence type according to the plurality of delimiting characters and the plurality of index values; Based on the data of the Token sequence type and the data of the Position sequence type, determine the embedding layer result of the metadata of each sample; The editor layer includes: multiple convolutional layers and a global multi-head attention layer; the second output module is specifically used for: The embedding layer results are input to the multiple convolutional layers of the editor layer to output multiple convolutional layer results; The results of the multiple convolutional layers are concatenated to obtain the final convolutional layer result; The results of the convolutional layer are input into the global multi-head attention layer of the editor layer to output the results of the global multi-head attention layer; The results of the global multi-head attention layer are determined as the editor layer results of the metadata of each sample.

7. A server, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the metadata processing method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the metadata processing method as described in any one of claims 1-5.

Images