A data directory matching optimization and generation method, system, device and medium

Abstract

The present invention proposes a data directory matching optimization and generation method, system, device and medium, which belong to the field of data processing technology. The method includes: obtaining text data to be managed, forming data to be optimized and resource data, and performing data preprocessing; using the SBERT model to vectorize the preprocessed data to generate sentence embedding vectors; by measuring the similarity between the sentence embedding vectors, identifying related content, and recommending data directory items based on the similarity measurement results; based on the recommended data directory items, quality verification and feedback are performed. The present invention realizes the improvement and optimization of the directory matching results of complex information systems by vectorizing the system's custom text data and data directory text and performing similarity calculation and intelligent recommendation through the SBERT model.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically to a data directory matching optimization and generation method, system, device and medium. Background Art

[0002] With the rapid development of information technology, all walks of life have accumulated unprecedented massive data resources in the wave of digital transformation. These data are not only a valuable basis for organizational decision-making, but also a key force in promoting business innovation and technological progress. However, faced with the real challenges of large data volume, wide variety, and high update frequency, the traditional manual management of data catalogs seems to be inadequate. It is not only inefficient and difficult to meet the ability to quickly respond to business needs, but also has many deficiencies in accuracy, which can easily cause data confusion and misunderstanding.

[0003] As a core component of data resource management, the accuracy and efficiency of data catalogs are directly related to data retrieval efficiency, analysis quality, and utilization value. An excellent data catalog can help users quickly locate the required information, improve the timeliness and accuracy of data processing, and provide strong support for the organization's decision support and business development.

[0004] However, the data resource directory chains of many organizations are currently facing many problems. The matching relationship between the directory data structure and the system data is chaotic, which makes it difficult to locate the data and find the required resources quickly. At the same time, the quality of the data directory is uneven, with a large amount of duplication, errors and missing information, which seriously affects the availability and credibility of the data. In addition, due to the complexity and diversity of text data, traditional matching methods are often difficult to deal with effectively, which greatly reduces the intelligence level of the data directory. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a data directory matching optimization and generation method, system, device and medium, which realizes the improvement and optimization of complex information system directory matching results by vectorizing system custom text data and data directory text and performing similarity calculation and intelligent recommendation through the SBERT model.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0007] In a first aspect, the present invention discloses a data directory matching optimization and generation method, comprising:

[0008] Obtain the text data to be managed, form the data to be optimized and resource data, and perform data preprocessing;

[0009] Use the SBERT model to vectorize the preprocessed data and generate the sentence embedding vector;

[0010] By measuring the similarity between the sentence embedding vectors, related content is identified and data catalog items are recommended based on the similarity measurement results;

[0011] Perform quality verification and feedback based on recommended data catalog items.

[0012] Further, the obtaining of the text data to be managed, forming the data to be optimized and the resource data, and performing data preprocessing, includes:

[0013] Obtain management regulations, business announcements, and custom text data to be managed from the organization's business information system as data to be optimized;

[0014] Generate resource data based on the existing data directory and the data resource directory to be refined;

[0015] Perform data cleaning on the data to be optimized and resource data.

[0016] Furthermore, the method further includes inputting the managed text data into a large model to generate an intelligent catalog. The specific process includes:

[0017] Perform semantic analysis on the managed text data and generate a relevant data resource directory based on the text data resources;

[0018] Conduct professional table analysis on relevant table data, and generate a unified data resource catalog for multiple data resources through multimodal understanding capabilities;

[0019] The generated data resource directory will be exported uniformly, the existing data resource directory will be supplemented, and it will continue to be improved.

[0020] Furthermore, the method further comprises:

[0021] Obtain data resources and related data directory chain text resources, and use the triplet loss function to train the SBERT model to improve the model's deep semantic feature capture ability;

[0022] The triplet loss function is:

[0023]

[0024] Among them, A is the embedding vector of the anchor sentence, P is the embedding vector of the positive sample sentence similar to the anchor sentence, N is the embedding vector of the negative sample sentence that is dissimilar to the anchor sentence, the function D represents the distance between the two vectors, and m is the margin parameter.

[0025] Furthermore, the SBERT model is used to vectorize the preprocessed data to generate an embedding vector of the sentence, including:

[0026] Based on the preprocessed data, use the word embedding model to extract the sentences and convert each word in the sentence into a word vector;

[0027] The word vector is input into the SBERT model, and the input word vector is processed through its multiple Transformer layers to generate the embedding vector of the sentence.

[0028] Furthermore, the SBERT model adopts a Siamese network structure and optimizes the model through the following loss function:

[0029]

[0030] in, Yes Input and The distance between the embedding vectors of is a label indicating whether the sample pair is similar.

[0031] Furthermore, the method of identifying related content by measuring the similarity between the embedding vectors of the sentences and recommending data directory items based on the similarity measurement results includes:

[0032] Sentence vectorization is performed using the following formula:

[0033] sentence_embedding = (word_embedding_1 + word_embedding_2 + ... +word_embedding_n) / n

[0034] Where n is the number of words in the sentence, word_embedding_i is the vector representation of the i-th word in the sentence;

[0035] Use the SBERT model to calculate the cosine similarity between the embedding vectors of the sentences in the data to be optimized and the embedding vectors of the sentences in the resource data;

[0036] The cosine similarity calculation formula used by the SBERT model is:

[0037]

[0038] in, and Represent the embedding vectors of the two sentences respectively, represents the dot product of two vectors, and Respectively represent the modulus lengths of the two embedded vectors;

[0039] For the calculated cosine similarity, the relevant text resources are determined according to a preset similarity threshold;

[0040] Based on the relevant text resources, the cosine similarity is sorted and the most matching text resources are selected as the recommended data catalog items.

[0041] Furthermore, the quality verification and feedback based on the recommended data catalog items include:

[0042] Based on the recommended data directory items, check the semantic consistency between the data items and the directories or subdirectories they are in, and use the SBERT model to calculate the similarity between the data items and the directory labels. If it is lower than the set threshold, it is considered an unreasonable match and feedback is given;

[0043] Verify that the key information of the data item is complete;

[0044] Use preset validation logic or external data sources to verify whether the specific information in the data item is accurate.

[0045] In a second aspect, the present invention further discloses a data directory matching optimization and generation system, comprising:

[0046] The data acquisition module is used to acquire the text data to be managed, form the data to be optimized and resource data, and perform data preprocessing;

[0047] The sentence embedding module is used to vectorize the preprocessed data using the SBERT model to generate sentence embedding vectors;

[0048] Intelligent recommendation module, used to identify relevant content by measuring the similarity between the embedding vectors of sentences, and recommend data directory items based on the similarity measurement results;

[0049] The verification and feedback module is used to perform quality verification and feedback based on the recommended data catalog items.

[0050] In a third aspect, the present invention further discloses a data directory matching optimization and generation device, comprising:

[0051] A memory for storing a data directory matching optimization and generation program;

[0052] A processor is used to implement the steps of the data directory matching optimization and generation method as described in any one of the above items when executing the data directory matching optimization and generation program.

[0053] In a fourth aspect, the present invention further discloses a readable storage medium, on which a data directory matching optimization and generation program is stored. When the data directory matching optimization and generation program is executed by a processor, the steps of the data directory matching optimization and generation method as described in any one of the above items are implemented.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1. By deeply understanding text sentences, the present invention can more accurately determine the similarity and correlation between keywords or phrases in the data resource directory and system data text, thereby improving the matching accuracy.

[0056] 2. The present invention can reduce the computational complexity of the model by optimizing algorithms such as self-attention mechanism and position encoding, so that it can maintain high efficiency when matching on large amounts of data. The SBERT model supports parallel computing and distributed training, and can make full use of hardware resources such as multi-core processors and GPUs to further improve the matching speed.

[0057] 3. The SBERT model used in the present invention is trained based on large-scale pre-training data, so it has strong generalization ability and can process data resource directories and system data texts in various fields and styles. Even when facing new or unknown data sets, the SBERT model can maintain good matching performance without retraining or adjusting model parameters.

[0058] 4. The present invention can be continuously optimized according to user feedback on the results, and can better meet user needs.

[0059] 5. The present invention can build a highly accurate, flexible and continuously improving intelligent question-answering system with intent recognition by comprehensively using technologies such as model training, intent recognition, data processing and continuous learning. It can not only improve user satisfaction, but also bring practical value to the organization, including reducing costs, improving efficiency and enhancing market competitiveness.

[0060] It can be seen that compared with the prior art, the present invention has outstanding substantive features and significant progress, and the beneficial effects of its implementation are also obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0062] Figure 1It is a method flow chart of a specific implementation mode of the present invention.

[0063] Figure 2 It is a flow chart of step S1 of a specific implementation mode of the present invention.

[0064] Figure 3 It is a flow chart of step S2 of a specific implementation mode of the present invention.

[0065] Figure 4 It is a flow chart of step S3 of a specific implementation mode of the present invention.

[0066] Figure 5 It is a flow chart of step S4 of a specific implementation mode of the present invention.

[0067] Figure 6 It is a system structure diagram of a specific implementation mode of the present invention.

[0068] Figure 7 It is a schematic diagram of the device structure of a specific embodiment of the present invention.

[0069] In the figure, 1. data acquisition module; 2. sentence embedding module; 3. intelligent recommendation module; 4. verification feedback module; 101. processor; 102. memory; 103. input interface; 104. output interface; 105. communication unit; 106. keyboard; 107. display; 108. mouse. DETAILED DESCRIPTION

[0070] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0071] See also Figure 1 As shown, this embodiment provides a data directory matching optimization and generation method, including the following steps:

[0072] S1: Obtain the text data to be managed, form the data to be optimized and resource data, and perform data preprocessing.

[0073] In a specific implementation, first, the management regulations, business announcements and custom text data to be governed are obtained from the organization's business information system as data to be optimized; at the same time, resource data is generated based on the existing data directory and the data resource directory to be refined. Then, the data to be optimized and the resource data are cleaned. In addition, in this step, the SBERT model is trained by obtaining data resources and related data directory chain text resources to improve the model's deep semantic feature capture capability.

[0074] It should be noted that this application can use the triplet loss function to train and optimize the SBERT model in a timely manner.

[0075] The triplet loss function can be defined as:

[0076]

[0077] Among them, A is the embedding vector of the anchor sentence, P is the embedding vector of the positive sample sentence similar to the anchor sentence, N is the embedding vector of the negative sample sentence that is dissimilar to the anchor sentence, D represents the distance function between the two vectors (such as Euclidean distance or cosine distance), and m is a margin parameter used to control the distance difference between positive and negative samples.

[0078] As an example, this step also includes: summarizing the directory name based on the managed text data using the big model. Specifically, the managed text data is obtained, and the corresponding data resources are summarized using the multimodal understanding capabilities of the relevant big model such as natural language and table language, and the summary data resource directory name is intelligently generated.

[0079] As an example, Figure 2 As shown in the figure, this step first obtains the first channel of data to be optimized from the custom text data such as management regulations and business announcements to be governed in the business information system of enterprises and institutions, and at the same time, forms the second channel of data resources based on the data directory text resources. Then the data is cleaned and stored in the governance database to prepare data for subsequent steps. In addition, through a large number of data resources and data directory chain text resources, the data model is trained to form a mature model that is more adaptable to the environment.

[0080] S2: Use the SBERT model to vectorize the preprocessed data and generate the sentence embedding vector.

[0081] In a specific implementation, firstly, based on the preprocessed data, a word embedding model is used to extract sentences therein, and each word in the sentence is converted into a word vector; then the word vector is input into the SBERT model, and the input word vector is processed through its multiple Transformer layers to generate an embedding vector for the sentence.

[0082] As an example, Figure 3 As shown in the figure, the purpose of this step is to convert each sentence into a high-dimensional vector through the SBERT model. This process is called sentence embedding.

[0083] Specifically, each word in a sentence is first converted into a word vector and then processed through multiple layers of the SBERT model to finally generate an embedding vector for the sentence.

[0084] The SBERT model uses a Siamese network structure, which can encode two sentences separately, generate fixed-size embeddings, and then fine-tune them through classification or cosine similarity objectives. This structure helps the model learn better sentence representations. Specifically, the model can be optimized through the following loss function:

[0085]

[0086] in, Yes Input and The distance between the embedding vectors of is a label indicating whether the sample pair is similar (1 for similarity and 0 for dissimilarity).

[0087] In addition, for specific tasks, such as text matching, the SBERT model can also be fine-tuned based on pre-training, using task-specific data to adjust the model's parameters to better suit the target task.

[0088] S3: Identify related content by measuring the similarity between the sentence embedding vectors, and recommend data directory items based on the similarity measurement results.

[0089] In a specific implementation, first, sentence vectorization is performed using the following formula:

[0090] sentence_embedding = (word_embedding_1 + word_embedding_2 + ... +word_embedding_n) / n;

[0091] Where n is the number of words in the sentence and word_embedding_i is the vector representation of the i-th word in the sentence.

[0092] Then, the SBERT model is used to calculate the cosine similarity between the embedding vector of the sentence of the data to be optimized and the embedding vector of the sentence of the resource data; the cosine similarity calculation formula used by the SBERT model is:

[0093]

[0094] in, and Represent the embedding vectors of the two sentences respectively, represents the dot product of two vectors, and Respectively represent the modulus length of the two embedding vectors. The value range of cosine similarity is [−1,1]. The larger the value, the more similar the two sentences are.

[0095] Furthermore, the calculated cosine similarity is used to determine relevant text resources according to a preset similarity threshold; finally, based on the relevant text resources, the cosine similarities are sorted and the most matching text resource is selected as the recommended data directory item.

[0096] As an example, Figure 4 As shown in the figure, this step processes sentence pairs, i.e., the combination of two sentences, through the SBERT model to evaluate the semantic similarity or association between them. The embeddings of the sentence pairs can be merged through simple vector operations (such as averaging, addition, etc.) to form a joint representation of the sentence pairs.

[0097] Among them, using sentence embedding, the SBERT model can calculate the similarity between sentences, usually through cosine similarity. Cosine similarity measures the degree of similarity between two vectors in direction. The closer the value is to 1, the more similar the sentences are. In the text matching task, the SBERT model compares the query sentence with the text in the directory. By calculating the similarity between the query sentence and each sentence in the directory, the most relevant text resources can be found.

[0098] In addition, according to the needs of the application scenario, a similarity threshold can be set to determine the matching standard. Only when the similarity exceeds this threshold, the two sentences are considered to be matched. Among multiple matching candidates, they are ranked according to the similarity and the most matching text resource is selected. It can be used to sort search results or provide the most relevant recommendations in the recommendation system.

[0099] S4: Perform quality verification and feedback based on the recommended data catalog items.

[0100] In a specific implementation, this step, on the one hand, checks the semantic consistency between the data item and its directory or subdirectory based on the recommended data directory item, and uses the SBERT model to calculate the similarity between the data item and the directory label. If it is lower than the set threshold, it is considered to be an unreasonable match and feedback is given; on the other hand, this step can verify whether the key information of the data item is complete; if it is incomplete, feedback is given. At the same time, this step also uses preset verification logic or external data sources to verify whether the specific information in the data item is accurate, and feedback is given if it is inaccurate.

[0101] As an example, Figure 5 As shown in the figure, first, check the semantic consistency between the data item and the directory or subdirectory it is in. Use the SBERT model to calculate the similarity between the data item and the directory label. If it is lower than the set threshold, it is considered an unreasonable match and prompts the administrator or user to pay attention.

[0102] In addition, verify that the key information of the data item is complete. For example, for a file data item, you may check whether the file name, description, tags, author, creation date, and other fields are missing.

[0103] At the same time, use preset verification logic or external data sources (such as databases, API interfaces) to verify whether the specific information in the data item is accurate.

[0104] It should be noted that this method supports both batch verification and incremental verification. Batch verification is used for comprehensive inspection after the initial construction or large-scale update of the data directory; incremental verification performs real-time or periodic inspections on newly added or modified data items to ensure continuous monitoring of data quality.

[0105] The present invention can provide an intuitive user feedback interface, allowing users to directly mark, comment or make modification suggestions for catalog items. The interface should be designed to be concise and clear, so that users can quickly understand and operate;

[0106] For reasonable adjustments and correction suggestions, data items can be moved to more appropriate directory locations, their descriptions or labels can be modified automatically or manually (depending on system configuration); and all feedback and processing results can be recorded for subsequent analysis and optimization.

[0107] The present invention can continuously identify problems in the data directory through automated verification and user feedback, and solve the problems through automatic adjustment or user correction; it can also continuously optimize its own verification rules and algorithms based on feedback results to improve the accuracy and efficiency of verification;

[0108] It can be seen that this method can promote the continuous optimization and development of data catalogs through the closed loop of data processing, and provide users with more efficient, accurate and convenient data services.

[0109] It can be seen that the present invention provides a data directory matching optimization and generation method, focusing on improving the accuracy of directory matching, and achieving the following effects:

[0110] (1) Accuracy: The SBERT model fine-tunes the BERT model to make it more suitable for sentence-level tasks, thereby being able to more accurately extract sentence-level semantic information. In text matching in data resource directories, this capability enables the system to more accurately determine whether two text resources express the same or similar semantic information, thereby significantly improving the matching accuracy.

[0111] (2) Matching efficiency: Compared with directly using the BERT model for text matching, the SBERT model greatly improves computing efficiency by generating sentence embeddings. In scenarios where a large amount of text resources need to be matched, the SBERT model can significantly reduce computing time and resource consumption, making the entire system more efficient.

[0112] (3) Utilization: By improving the accuracy and efficiency of text matching, this method helps optimize the resource utilization in the data resource directory chain. This method can more accurately identify and match the text resources required by users, reduce invalid searches and duplication of work, and improve overall work efficiency.

[0113] (4) Scalability: The SBERT model has good scalability and can be easily integrated into various data resource directory systems. In addition, with the continuous optimization and updating of the SBERT model, this optimization method can also continuously improve its performance to adapt to more complex and diverse matching requirements.

[0114] (5) Improve user experience: As the accuracy and efficiency of text matching are improved, users can find the information resources they need more quickly when using the data resource directory system, thereby improving user experience. Improved user satisfaction will help enhance the user stickiness and market competitiveness of the system.

[0115] See also Figure 6 As shown, the present invention also discloses a data directory matching optimization and generation system, including: a data acquisition module 1, a sentence embedding module 2, an intelligent recommendation module 3 and a verification feedback module 4.

[0116] The data acquisition module 1 is used to acquire the text data to be managed, form the data to be optimized and the resource data, and perform data preprocessing.

[0117] Sentence embedding module 2 is used to vectorize the preprocessed data using the SBERT model to generate sentence embedding vectors.

[0118] The intelligent recommendation module 3 is used to identify related content by measuring the similarity between the embedding vectors of the sentences, and recommend data directory items based on the similarity measurement results.

[0119] The verification and feedback module 4 is used to perform quality verification and feedback based on the recommended data catalog items.

[0120] The specific implementation of the data directory matching optimization and generation system of this embodiment is basically the same as the specific implementation of the data directory matching optimization and generation method described above, and will not be repeated here.

[0121] See also Figure 7 As shown, the present invention also discloses a data directory matching optimization and generation device, including a processor 101 and a memory 102; wherein, when the processor 101 executes the data directory matching optimization and generation program stored in the memory, the steps of the data directory matching optimization and generation method as described in any one of the above items are implemented.

[0122] The data directory matching optimization and generation device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer or a desktop computer.

[0123] Among them, the processor 101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 101 can be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 101 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 101 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0124] The memory 102 may include one or more computer-readable storage media, which may be non-transitory. The memory 102 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 102 is at least used to store the following computer program, wherein, after the computer program is loaded and executed by the processor 101, the relevant steps of the data directory matching optimization and generation method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 102 may also include an operating system and data, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system may include Windows, Unix, Linux, etc. The data may include, but is not limited to, the data involved in the above-mentioned data directory matching optimization and generation method, etc.

[0125] Furthermore, the data directory matching optimization and generation device in this embodiment may also include:

[0126] The input interface 103 is used to obtain the data directory matching optimization and generation program imported from the outside, and save the obtained data directory matching optimization and generation program to the memory 102. It can also be used to obtain various instructions and parameters transmitted by external terminal devices, and transmit them to the processor 101, so that the processor 101 can use the above various instructions and parameters to carry out corresponding processing. In this embodiment, the input interface 103 can specifically include but is not limited to a USB interface, a serial interface, a voice input interface, a fingerprint input interface, a hard disk reading interface, etc.

[0127] The output interface 104 is used to output various data generated by the processor 101 to a terminal device connected thereto, so that other terminal devices connected to the output interface can obtain various data generated by the processor 101. In this embodiment, the output interface 104 may specifically include but is not limited to a USB interface, a serial interface, and the like.

[0128] The communication unit 105 is used to establish a remote communication connection between the server operation service optimization configuration device and the external server, so that the data directory matching optimization and generation device can mount the image file to the external server. In this embodiment, the communication unit 105 can specifically include but is not limited to a remote communication unit based on wireless communication technology or wired communication technology.

[0129] The keyboard 106 is used to obtain various parameter data or instructions input by the user by tapping the keycaps in real time.

[0130] The display 107 is used to display the relevant information of the data directory matching optimization and generation process in real time.

[0131] The mouse 108 can be used to assist the user in inputting data and simplify the user's operation.

[0132] The present invention also discloses a readable storage medium, wherein the readable storage medium includes a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable hard disk, a CD-ROM or any other form of storage medium known in the technical field. The readable storage medium stores a data directory matching optimization and generation program, and when the data directory matching optimization and generation program is executed by a processor, the steps of the data directory matching optimization and generation method described in any one of the above items are implemented.

[0133] In summary, the present invention realizes the improvement and optimization of the matching results of the complex information system directory by vectorizing the system custom text data and data directory text and performing similarity calculation and intelligent recommendation through the SBERT model.

[0134] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the method disclosed in the embodiment, since it corresponds to the system disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0135] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0136] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0137] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit.

[0139] Similarly, each processing unit in each embodiment of the present invention may be integrated into one functional module, or each processing unit may exist physically, or two or more processing units may be integrated into one functional module.

[0140] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0141] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0142] The data directory matching optimization and generation method, system, device and readable storage medium provided by the present invention are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A data directory matching optimization and generation method, characterized in that: include: Obtain the text data to be managed, form the data to be optimized and resource data, and perform data preprocessing; Use the SBERT model to vectorize the preprocessed data and generate the sentence embedding vector; By measuring the similarity between the sentence embedding vectors, related content is identified and data catalog items are recommended based on the similarity measurement results; Perform quality verification and feedback based on recommended data catalog items; The SBERT model is used to vectorize the preprocessed data to generate an embedding vector of the sentence, including: Based on the preprocessed data, use the word embedding model to extract the sentences and convert each word in the sentence into a word vector; The word vector is input into the SBERT model, and the input word vector is processed through its multiple Transformer layers to generate the sentence embedding vector; The SBERT model adopts the Siamese network structure and optimizes the model through the following loss function: in, Yes Input and The distance between the embedding vectors of is a label indicating whether the sample pair is similar; The method of measuring the similarity between the embedding vectors of the sentences, identifying the related content, and recommending the data directory items based on the similarity measurement results includes: Sentence vectorization is performed using the following formula: sentence_embedding = (word_embedding_1 + word_embedding_2 + ... + word_embedding_n) / n; Where n is the number of words in the sentence, word_embedding_i is the vector representation of the i-th word in the sentence; Use the SBERT model to calculate the cosine similarity between the embedding vectors of the sentences in the data to be optimized and the embedding vectors of the sentences in the resource data; The cosine similarity calculation formula used by the SBERT model is: in, and Represent the embedding vectors of the two sentences respectively, represents the dot product of two vectors, and Respectively represent the modulus lengths of the two embedded vectors; For the calculated cosine similarity, the relevant text resources are determined according to a preset similarity threshold; Based on the relevant text resources, the cosine similarity is sorted and the most matching text resources are selected as the recommended data catalog items.

2. The data directory matching optimization and generation method according to claim 1, characterized in that: The step of obtaining the text data to be managed, forming the data to be optimized and the resource data, and performing data preprocessing includes: Obtain management regulations, business announcements and custom text data to be managed from the business information systems of various organizations as data to be optimized; Generate resource data based on the existing data directory and the data resource directory to be refined; Perform data cleaning on the data to be optimized and resource data.

3. The data directory matching optimization and generation method according to claim 1, characterized in that: The method further comprises: Obtain data resources and related data directory chain text resources, and use the triplet loss function to train the SBERT model to improve the model's deep semantic feature capture ability; The triplet loss function is: Among them, A is the embedding vector of the anchor sentence, P is the embedding vector of the positive sample sentence similar to the anchor sentence, N is the embedding vector of the negative sample sentence that is dissimilar to the anchor sentence, the function D represents the distance between the two vectors, and m is the margin parameter.

4. The data directory matching optimization and generation method according to claim 1, characterized in that: The quality verification and feedback based on the recommended data catalog items include: Based on the recommended data directory items, check the semantic consistency between the data items and the directories or subdirectories they are in, and use the SBERT model to calculate the similarity between the data items and the directory labels. If it is lower than the set threshold, it is considered an unreasonable match and feedback is given; Verify that the key information of the data item is complete; Use preset validation logic or external data sources to verify whether the specific information in the data item is accurate.

5. A data directory matching optimization and generation system, characterized in that: include: The data acquisition module is used to acquire the text data to be managed, form the data to be optimized and resource data, and perform data preprocessing; The sentence embedding module is used to vectorize the preprocessed data using the SBERT model to generate sentence embedding vectors; Intelligent recommendation module, used to identify relevant content by measuring the similarity between the embedding vectors of sentences, and recommend data directory items based on the similarity measurement results; Verification and feedback module, used to perform quality verification and feedback based on recommended data catalog items; The SBERT model is used to vectorize the preprocessed data to generate an embedding vector of the sentence, including: Based on the preprocessed data, use the word embedding model to extract the sentences and convert each word in the sentence into a word vector; The word vector is input into the SBERT model, and the input word vector is processed through its multiple Transformer layers to generate the sentence embedding vector; The SBERT model adopts the Siamese network structure and optimizes the model through the following loss function: in, Yes Input and The distance between the embedding vectors of is a label indicating whether the sample pair is similar; The method of measuring the similarity between the embedding vectors of the sentences, identifying the related content, and recommending the data directory items based on the similarity measurement results includes: Sentence vectorization is performed using the following formula: sentence_embedding = (word_embedding_1 + word_embedding_2 + ... + word_embedding_n) / n; Where n is the number of words in the sentence, word_embedding_i is the vector representation of the i-th word in the sentence; Use the SBERT model to calculate the cosine similarity between the embedding vectors of the sentences in the data to be optimized and the embedding vectors of the sentences in the resource data; The cosine similarity calculation formula used by the SBERT model is: in, and Represent the embedding vectors of the two sentences respectively, represents the dot product of two vectors, and Respectively represent the modulus lengths of the two embedded vectors; For the calculated cosine similarity, the relevant text resources are determined according to a preset similarity threshold; Based on the relevant text resources, the cosine similarity is sorted and the most matching text resources are selected as the recommended data catalog items.

6. A data directory matching optimization and generation device, characterized in that: include: A memory for storing a data directory matching optimization and generation program; A processor is used to implement the steps of the data directory matching optimization and generation method as described in any one of claims 1 to 4 when executing the data directory matching optimization and generation program.

7. A readable storage medium, characterized in that: The readable storage medium stores a data directory matching optimization and generation program, which, when executed by a processor, implements the steps of the data directory matching optimization and generation method as claimed in any one of claims 1 to 4.

Images