Data sharing open platform data resource searching method and system based on large language model

Abstract

The invention discloses a data sharing open platform data resource search method and system based on a large language model, which utilize strong semantic understanding and logical reasoning ability of the large language model to automatically analyze data requirements in a natural language form provided by a data user, and can obtain an accurate query result without professional knowledge of the user. The big language model can better capture the intention behind the data demand, so that the data resource meeting the demand can be more accurately positioned. In combination with factors such as data consanguinity and access time, the system can be associated with other potential related data resources to form a wider target data resource set. According to the method, not only is the search breadth improved, but also important data resources are ensured not to be omitted, and the success rate of data search and the user experience are greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a method and system for searching data resources on a data sharing open platform based on a large language model. Background Art

[0002] With the advent of the big data era, data sharing and open platforms have become an important infrastructure for promoting cross-departmental data circulation and utilization. These platforms are designed to achieve data collection, governance, sharing and application among different government departments to improve work efficiency and service quality. When data users (i.e. data users) find the data resources they need on the platform, they usually rely on precise search or fuzzy search to provide keywords to match data resource names or other metadata fields. When the data user finds the right data resource, it is also necessary to submit an application to the data supplier and obtain approval before data docking can be carried out.

[0003] However, there is a significant technical problem with existing search methods: it is difficult for data users to accurately describe their data needs. Due to the large number of data resources, complex naming, and the fact that the names of some data resources cannot directly reflect their business meanings, data users often need to have a certain understanding of the list of data resources on the data sharing open platform in order to provide appropriate query content. This not only increases the usage threshold for data users, but also limits the scope of data they can obtain. In addition, the manual analysis of data needs is inefficient, making it difficult to quickly respond to a large number of data requests, and is limited by the knowledge scope of analysts, and may not be able to fully cover all relevant data resources. Summary of the invention

[0004] The purpose of the present invention is to provide a data resource search method and system for a data sharing and open platform based on a large language model, which fully utilizes various information about data resources on the data sharing and open platform, effectively improves the depth and breadth of data users' query of data resources, and helps data users quickly locate the required data resources to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides a method for searching data resources on a data sharing open platform based on a large language model, comprising the following steps: Pre-process all data resources of the data sharing and open platform, convert data resource names, classification information, label information, business description information and data fields into vector representations through embedded models, and store them in the vector database; Collect the data requirements of the user, set prompts and input them into the large language model, analyze and extract the relevant information of the data resources required to solve the requirements; Vectorize the data resource related information obtained from the large language model and retrieve the matching data resources from the vector database formed by the preprocessing module; According to the data lineage relationship of the retrieved data resources and the access time of the data provider, other related data resources are associated to form a target data resource set; The actual data of the target data resource set is vectorized and compared with the vector of the required data resource information to find the most similar actual data sample.

[0006] Preferably, the embedded model adopts the Chinese-friendly pre-trained model M3E-base.

[0007] Preferably, the large language model is ChatGLM, and the prompt is designed in accordance with the CoT thinking chain.

[0008] Preferably, the number of data resources matched in the retrieval is limited to a maximum of five.

[0009] Preferably, the expansion of data resources is achieved by traversing the data lineage network of the retrieved data resources, adding adjacent data resources and other data resources accessed by the same data provider on the same day.

[0010] Preferably, the number of actual data samples returned is limited to the five most similar data for each data resource.

[0011] Another aspect of the present invention provides a data resource search system for a data sharing open platform based on a large language model, comprising: The data resource preprocessing module is used to preprocess the full amount of data resources of the data sharing and open platform. It converts the data resource name, classification information, label information, business description information and data fields into vector representation through the embedded model and stores them in the vector database. The data resource retrieval module is used to collect the data requirements of the data user, set prompts and input them into the large language model, analyze and extract the relevant information of the data resources required to solve the requirements, vectorize the relevant information of the data resources obtained from the large language model, and retrieve matching data resources in the vector database formed by the preprocessing module; The extended data resource module is used to associate other related data resources according to the data lineage relationship of the retrieved data resources and the access time of the data supply party to form a target data resource set; The actual data retrieval module performs vectorization processing on the actual data of the target data resource set, compares it with the vector of the required data resource information, and finds the most similar actual data sample.

[0012] Vector database, used for persistent storage of data resources and vector representation of their actual data.

[0013] Preferably, the data resource retrieval module is capable of processing data requirements in natural language form, automatically extracting and matching corresponding data resource features by analyzing the semantic content of the data requirements, and achieving accurate search without the need for the user to have a comprehensive understanding of the data resource list.

[0014] Preferably, the actual data retrieval module performs data desensitization before returning the target data resource list and the corresponding data samples to ensure that sensitive information is not leaked and that the returned data samples meet the requirements of privacy protection.

[0015] Technical effects and advantages of the present invention: Compared with the prior art, the present invention provides a data resource search method and system for a data sharing open platform based on a large language model, which has the following advantages: The present invention utilizes the powerful semantic understanding and logical reasoning capabilities of the large language model to automatically parse the data requirements in the natural language form provided by the user, and obtain accurate query results without the user having professional knowledge. The large language model can better capture the intention behind the data requirements, thereby more accurately locating the data resources that meet the requirements. Combined with factors such as data kinship and access time, the system can associate other potential related data resources to form a more extensive set of target data resources. This method not only increases the breadth of the search, but also ensures that important data resources are not missed. By vectorizing the actual data samples of the target data resources and comparing them with the required data resource information, the system can find the most similar data samples, helping the user to intuitively judge whether the data resources meet their needs, greatly improving the success rate of data search and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flow chart of a method for searching data resources on a data sharing open platform based on a large language model according to the present invention; Figure 2 The present invention is a block diagram of a data resource search system for a data sharing open platform based on a large language model. DETAILED DESCRIPTION

[0017] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0018] The present invention provides a data resource search method for a data sharing open platform based on a large language model, which uses a large language model to perform semantic analysis on data needs to form data resources required to solve data needs, and uses a large model combined with a series of metadata such as data grading and classification information, data business description, data lineage, etc. collected by the data sharing open platform when aggregating data resources to compare, find out several data resources related to the data needs, and then use the large model to perform similarity comparison between the required data resources and the actual data of each related data resource to find the most relevant data samples. It makes full use of various information about data resources on the data sharing open platform, effectively improves the depth and breadth of data users' query of data resources, and helps data users quickly locate the required data resources. The details are as follows: like Figure 1 As shown, a data resource search method for a data sharing open platform based on a large language model in this embodiment includes the following steps: Step 1: Preprocess all data resources of the data sharing and open platform, convert the data resource name, classification information, label information, business description information and data field into vector representation through the embedded model, and store them in the vector database; further, the embedded model adopts the Chinese-friendly pre-trained model M3E-base.

[0019] Specifically, in step 1, all data resources of the data sharing open platform are preprocessed, and the data resource name, classification information, label information, business description information and data fields are converted into vector representations by using the Chinese-friendly pre-trained embedded model M3E-base, and then stored in the vector database. This process brings the following significant technical effects: Improve search efficiency: By converting relevant information of all data resources into vector representations and storing them in a vector database in advance, when users make data demands, they can quickly retrieve the closest data resources in high-dimensional space, thereby greatly improving the search speed and response time.

[0020] Enhanced semantic understanding: Using Chinese-friendly pre-trained models such as M3E-base for vectorization can better capture the semantic information in Chinese text. This enables the system to more accurately understand and match the actual meaning of data resources, rather than relying solely on keywords or superficial similarities, thereby improving the relevance and accuracy of search results.

[0021] Support for complex queries: Since vector representation can express richer semantic information, this method not only supports simple keyword-based searches, but also complex natural language queries. For example, a user can describe their needs in natural language instead of just providing a few keywords, and the system can still effectively find matching data resources.

[0022] Expansion adaptability: The M3E-base model has good generalization capabilities. For newly added data resources or data from different fields, better vectorization effects can be obtained without retraining the model, ensuring the flexibility and scalability of the system.

[0023] Optimize resource management: By converting the information of data resources into a vector representation in a unified format and storing it centrally, it is convenient for subsequent data management and maintenance. At the same time, it is also convenient to interact with other modules (such as the retrieval module) to ensure the efficient operation of the entire system.

[0024] In summary, the technical solution of preprocessing data resources in step one and using the M3E-base model for vectorized processing provides a solid foundation for subsequent data resource searches and significantly improves the efficiency, accuracy and user experience of data search.

[0025] Step 2: Collect the data requirements of the user, set prompts and input them into the large language model, analyze and extract relevant information of the data resources required to solve the requirements; further, the large language model is ChatGLM, and the design of the prompts follows the CoT thinking chain.

[0026] In step 2, we collect the data requirements of users, set prompts written in accordance with the CoT (Chain of Thought), and input this information into the large language model ChatGLM to analyze and extract relevant information about the data resources required to solve the requirements. This process brings the following significant technical effects: Improve the accuracy of natural language understanding: As an advanced large language model, ChatGLM has powerful natural language processing capabilities and can deeply understand the complex data requirements raised by users. Combined with the prompts designed in the CoT method, the model can reason step by step, not only focusing on the surface keyword matching, but also capturing the deep semantics and logical relationships, thereby providing more accurate information about data resources.

[0027] Enhance query flexibility and intelligence: By using a large language model for demand analysis, the system is no longer limited to the traditional keyword search mode, but can understand and respond to more complex natural language queries. This provides users with a more flexible and intelligent interactive experience, and even users who are unfamiliar with the platform can obtain accurate data resource recommendations through natural language expressions.

[0028] Optimize the matching accuracy of data resources: The prompt design of the CoT thinking chain guides the ChatGLM model to think about problems step by step according to the logical chain, analyze the needs of the data users from different angles, and ensure that no important information points are missed. This method helps to discover implicit data needs, thereby improving the accuracy and comprehensiveness of the data resources finally matched.

[0029] Promoting human-machine collaboration efficiency: Data users can easily express their needs through natural language without having to deeply understand the specific details of the data sharing open platform or have professional query skills. The large language model plays the role of an intelligent assistant, quickly and accurately analyzing needs and providing corresponding data resource suggestions, greatly shortening the time cycle from raising needs to obtaining data and improving overall work efficiency.

[0030] Support continuous learning and improvement: Large language models such as ChatGLM have a certain degree of adaptive learning ability. As more data requirements in real scenarios are input, the model can continuously optimize its understanding and reasoning capabilities. This means that the system can become more and more intelligent over time and better meet the diverse needs of users.

[0031] In summary, step 2 achieves a deep understanding and efficient processing of the data needs of users by adopting the advanced large language model ChatGLM and the carefully designed CoT thinking chain prompt, significantly enhancing the flexibility, intelligence level and matching accuracy of data resource search, and greatly improving user experience and work efficiency.

[0032] Step 3: Vectorize the data resource related information obtained from the large language model, and retrieve matching data resources from the vector database formed by the preprocessing module; the number of matching data resources retrieved is limited to a maximum of five.

[0033] In step three, the data resource related information obtained from the large language model is vectorized, and matching data resources are retrieved from the vector database formed by the preprocessing module, and the number of matching data resources retrieved is limited to a maximum of five. This process brings the following significant technical effects: Improve search efficiency and response speed: By converting relevant information about data resources into vector representation, the system can quickly calculate similarities in high-dimensional space and retrieve the closest matches. This not only speeds up the search, but also ensures that results can be given quickly even when faced with massive data resources, greatly improving the system's response speed and user experience.

[0034] Enhanced matching accuracy: Vectorized processing can capture the semantic features in data resource information, rather than just surface matching based on keywords. This means that the system can more accurately understand the needs of users, find data resources that truly meet the needs, reduce false positives or irrelevant results, and thus improve the accuracy of matching.

[0035] Optimize user decision support: Limiting the number of search results to a maximum of five avoids the confusion that too many choices bring to users. This design helps simplify the decision-making process of users, allowing them to focus more on evaluating a small number of high-quality candidate data resources rather than screening through a large number of potentially irrelevant options. At the same time, this also reduces unnecessary information overload and improves user satisfaction.

[0036] Promoting efficient resource utilization: Limiting the number of search results to five enables the system to focus more on returning the data resources that are most likely to meet the needs, thereby maximizing the value of each query. This approach not only improves the utilization of data resources, but also reduces the time cost of users exploring irrelevant data.

[0037] Support continuous optimization and feedback mechanism: Each search returns a limited number of results, which facilitates the establishment of an efficient feedback loop. Using data can more easily provide feedback on the relevance of search results, helping the system to continuously improve its vectorization processing and retrieval algorithms to better adapt to changes in demand in actual application scenarios.

[0038] Enhance system explainability: Limiting the number of output results makes the results of each query more controllable and explainable. For each recommended data resource, the system can provide detailed matching reasons or paths, which increases transparency and makes it easier for users to understand and trust the search results.

[0039] In summary, step 3 not only improves search efficiency and matching accuracy by vectorizing data resource related information and retrieving matching items (limited to a maximum of five) in the vector database, but also optimizes the user's decision support experience, promotes efficient resource utilization, and supports continuous optimization of the system. These technical effects work together to significantly enhance the effectiveness and practicality of the data resource search method of the data sharing open platform.

[0040] Step 4: According to the data lineage relationship of the retrieved data resources and the access time of the data supplier, other related extended data resources are associated to form a target data resource set; the extended data resources are achieved by traversing the data lineage network of the retrieved data resources, adding adjacent data resources and other data resources accessed by the same data supplier on the same day.

[0041] In step 4, other related extended data resources are associated according to the data lineage relationship of the retrieved data resources and the access time of the data supplier to form a target data resource set. Specifically, this process is achieved by traversing the data lineage network of the retrieved data resources, adding adjacent data resources and other data resources accessed by the same data supplier on the same day. This method brings the following significant technical effects: Expand your search: By associating related data resources through data lineage and the access time of data providers, the system can discover and introduce more potential related resources. This not only expands the coverage of search results, but also ensures that data resources that are closely related to the original search results but not directly matched are not missed, thereby improving the comprehensiveness and depth of the search.

[0042] Improving the relevance of data resources: Data lineage reflects the upstream and downstream dependencies between data resources, and this association usually means that there is an intrinsic connection or common background between data resources. Therefore, the extended data resources introduced based on data lineage are more likely to meet the needs of data users, improving the relevance and practicality of recommendation results.

[0043] Enhance the consistency of data resources: Other data resources accessed by the same data supplier on the same day often belong to the same project or batch and have a high degree of coherence and consistency. Including these data resources in the target set can help data users obtain a more complete and systematic data view, avoid information fragmentation, and promote comprehensive analysis and application.

[0044] Optimize user exploration experience: Expanding data resources provides users with an opportunity to explore more possibilities, not just limited to the initial search results. Users can take this opportunity to discover new data resources that may provide additional value for their needs. This approach enriches the user's interactive experience and promotes deeper data mining and utilization.

[0045] Improve data sharing efficiency: By automatically associating and expanding related data resources, the time cost of manual search and comparison by users is reduced, and the overall efficiency of data sharing is improved. At the same time, this also reduces the risk of errors caused by manual operations and ensures the accuracy and reliability of data resource association.

[0046] Support data analysis in complex scenarios: In some complex application scenarios, a single data resource may not be sufficient to meet the needs of the user. By introducing related data resources, the system can provide users with a richer data set, support more complex and multi-dimensional data analysis tasks, and enhance the service capabilities of the platform.

[0047] In summary, step 4 effectively expands the search scope of data resources, improves the relevance and coherence of recommendation results, optimizes the user's exploration experience, and promotes efficient data sharing and data analysis in complex scenarios through the association mechanism based on data lineage and data supply party access time. These technical effects work together to significantly enhance the practicality and user experience of the data resource search method of the data sharing open platform.

[0048] Step 5: Vectorize the actual data of the target data resource set, compare it with the vector of the required data resource information, find the most similar actual data samples, and limit the number of actual data samples returned to the five most similar data for each data resource.

[0049] In step 5, the actual data of the target data resource set is vectorized and compared with the vector of the required data resource information to find the most similar actual data samples, and the number of actual data samples returned is limited to the five most similar data for each data resource. This process brings the following significant technical effects: Improve matching accuracy and relevance: By vectorizing the actual data in the target data resource set and comparing it with the vector of the required data resource information, the system can accurately calculate the semantic similarity between the two. This not only improves the accuracy of the match, but also ensures that the returned data samples are highly relevant to the needs of the user, reducing the interference of irrelevant or low-quality results.

[0050] Optimize user experience: The number of actual data samples returned is limited to the five most similar to each data resource, avoiding the confusion of too many choices for users. This approach allows users to focus more on evaluating a small number of high-quality data samples, simplifies the decision-making process, and improves user experience and satisfaction.

[0051] Facilitates rapid verification of requirements: Returning the most similar data samples enables users to intuitively see the actual data content and quickly determine whether the data meets their needs. This instant feedback mechanism helps users quickly verify the relevance and applicability of data resources, reducing the time cost of subsequent communication and confirmation.

[0052] Enhance the transparency and explainability of the system: By providing specific data examples, the system increases the transparency of recommendation results. Users can clearly understand why certain data resources are considered the most relevant, and can further propose more specific requirements or adjust query conditions based on these examples. This enhances users' trust in the system's recommendations.

[0053] Support efficient data exploration and utilization: Actual data samples provide a direct view of the data, helping users better understand the data structure and content, and promoting data exploration and in-depth analysis. Users can discover new application scenarios or potential value based on these samples and promote the effective use of data.

[0054] Reduce information overload: Limiting the number of data samples returned to five effectively controls the amount of information and prevents information overload. This approach ensures that users are not overwhelmed by a large amount of data, but are able to make informed choices based on limited but high-quality information.

[0055] Improve system performance and efficiency: Vectorization and similarity calculation are performed in a high-dimensional space, which is more efficient than traditional text matching methods. At the same time, limiting the number of results returned also helps reduce computational complexity and improve the overall performance and response speed of the system.

[0056] In summary, step five finds the most similar data samples (the five most similar ones for each data resource) by vectorizing the actual data of the target data resource set and comparing it with the required data resource information, which significantly improves the matching accuracy, optimizes the user experience, promotes rapid demand verification, enhances the transparency of the system, and supports efficient data exploration and utilization.

[0057] On the other hand, this implementation proposes a data resource search system for a data sharing open platform based on a large language model, such as Figure 2 As shown, including: The data resource preprocessing module is used to preprocess the full amount of data resources of the data sharing and open platform. It converts the data resource name, classification information, label information, business description information and data fields into vector representation through the embedded model and stores them in the vector database. Specifically, the data resource preprocessing module uses an embedded model to pre-vectorize the data resource names, data classification information, data label information, business description information, and data fields of the full amount of data resources on the data sharing and open platform to form a data resource vector database. The information required by this module is based on the data sharing and open platform, so in order to improve the efficiency of the search, the operations described in this module occur before the data demand is generated.

[0058] Through the Chinese-friendly pre-trained embedded model M3E-base, the data resource names, data classification information, data label information, business description information, and data fields of the full data resources of the data sharing and open platform are vectorized in advance. After the vectorization operation, each data resource is mapped into a vector that can be stored and understood by a computer, and the formed series of vectors are persistently stored in the vector database Chroma to prepare for the next step of retrieval. The information required by this module is based on the data sharing and open platform, so in order to improve the efficiency of the search, the operations described in this module occur before the data demand is generated.

[0059] The data resource retrieval module is used to collect the data requirements of the data user, set prompts and input them into the large language model, analyze and extract the relevant information of the data resources required to solve the requirements, vectorize the relevant information of the data resources obtained from the large language model, and retrieve matching data resources in the vector database formed by the preprocessing module; Specifically, the data resource retrieval module collects the data requirements of the user, sets prompts for the data requirements, inputs them into the large language model for analysis, extracts the data classification, data description, and data field list information of the data resources required to solve the data requirements, vectorizes the data classification, data description, and data field list information of the required data resources, and then retrieves up to 5 data resources from the data resource vector database formed by the data resource preprocessing module.

[0060] By utilizing the good logical reasoning ability of the large language model, we can understand the data requirements, extract various information about the required data resources, and then search the pre-generated vector database to efficiently find matching data resources. Data resource retrieval module: This module collects the data requirements of the user, sets prompts for the data requirements, inputs them into the ChatGLM large language model for analysis, and extracts the data classification, data description, and data field list information of the data resources required to solve the data requirements.

[0061] The design of the prompt follows the CoT thinking chain approach, and guides and rules are designed to improve the ability to understand large models. The specific prompt format is as follows: "You are a data demand analyst who is good at using data resources to solve practical needs. Now let's think step by step about what data resources may be used to solve the needs of xxx? Please list the data name, data classification, data description, and main fields of several data resources respectively."

[0062] Next, the data classification, data description, and data field list information of the required data resources formed by the answer of the large model need to be vectorized by the embedded model, and then similarity is calculated in the data resource vector database formed by the data resource preprocessing module to retrieve up to 5 data resources. This module uses the good logical reasoning ability of the large language model to understand the data requirements, complete the extraction of various information about the required data resources, and then search the pre-generated vector database to efficiently find matching data resources.

[0063] The extended data resource module is used to associate other related data resources according to the data lineage relationship of the retrieved data resources and the access time of the data supply party to form a target data resource set; Specifically, the expansion data resource module includes a traversal retrieval data resource module, which is used to form several data resources, find the data resource with the closest data relationship according to the data lineage relationship of each data resource, and add the retrieved data resource; according to the time when the user of each data resource provides the current data resource, find out the other data resources accessed by the supplier of each data resource on the day of accessing the data, add the retrieved data resource to form the target data resource. Fully sharing the data lineage information and data source information about data resources on the open platform can effectively increase the breadth of the data resources found.

[0064] In the process of sharing, subscribing, gathering, and governing data resources on the data sharing and open platform, upstream and downstream relationships between data resources will be formed, thus forming a data lineage network. In the data lineage network, traverse and retrieve several data resources formed by the data resource module, find adjacent data resources, and add the retrieved data resources.

[0065] The data resources on the data sharing and open platform will record the data access time. When the same unit attaches data resources, it will also attach a whole set of related data resources. Therefore, it is necessary to find out the other data resources that the supplier of each data resource accessed on the day of data access based on the time when the user of each data resource provided the current data resource, and add the retrieved data resources to form the target data resource. Fully utilizing the data lineage information and data source information about data resources on the data sharing and open platform can effectively increase the breadth of the data resources found.

[0066] The actual data retrieval module vectorizes the actual data of the target data resource set, compares it with the vector of the required data resource information, and finds the most similar actual data sample. The data resource retrieval module can process data requirements in natural language form, automatically extract and match the corresponding data resource features by analyzing the semantic content of the data requirements, and can achieve accurate search without the need for a comprehensive understanding of the data resource list. Before returning the target data resource list and the corresponding data sample, the actual data retrieval module first performs data desensitization processing to ensure that sensitive information will not be leaked, and at the same time ensure that the returned data sample meets the requirements of privacy protection.

[0067] Specifically, the actual data retrieval module uses the embedded model again to vectorize the actual data of the target data resources formed by the extended data resource module to form a target data resource vector database; the required data resource information formed by the retrieval data resource module is vectorized and compared with the actual data of each data resource in the target data resource vector database, and the five most similar actual data of each data resource are retrieved, desensitized to form data samples, and the target data resource list and the corresponding data samples are returned.

[0068] By returning matching data samples, users can quickly understand the matching status of target data resources with the data requirements. Retrieve actual data module: The open source embedded model M3E-base supports Chinese and English text processing, and performs slightly better than text2vec and openai-ada embedded models in Chinese text similarity and text retrieval tasks.

[0069] This module uses the Chinese-friendly pre-trained embedded model M3E-base to vectorize the actual data of the target data resources formed by the expansion data resource module, and persistently stores the formed series of vectors in the vector database Chroma; the vectors of the required data resource information formed by the retrieval data resource module are compared with the actual data vectors of each data resource in the target data resource vector database, and the cosine similarity is calculated. The 5 most similar actual data of each data resource are retrieved, desensitized to form data samples, and the target data resource list and the corresponding data samples are returned. By returning the matching data samples, the user can quickly understand the matching situation between the target data resources and the proposed data requirements.

[0070] In summary, the present invention utilizes the powerful semantic understanding and logical reasoning capabilities of the large language model to automatically parse the data requirements in the natural language form provided by the user, and obtains accurate query results without the user having professional knowledge.

[0071] Large language models can better capture the intent behind data requirements, thereby more accurately locating data resources that meet the requirements. Combining factors such as data lineage and access time, the system can associate other potential related data resources to form a wider set of target data resources. This approach not only increases the breadth of the search, but also ensures that important data resources are not missed.

[0072] By vectorizing the actual data samples of the target data resources and comparing them with the required data resource information, the system can find the most similar data samples, helping data users to intuitively determine whether the data resources meet their needs, greatly improving the success rate of data search and user experience.

[0073] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data resource search method for a data sharing open platform based on a large language model, characterized in that: The following steps are involved: Pre-process all data resources of the data sharing and open platform, convert data resource names, classification information, label information, business description information and data fields into vector representations through embedded models, and store them in the vector database; Collect the data requirements of the user, set prompts and input them into the large language model, analyze and extract the relevant information of the data resources required to solve the requirements; Vectorize the data resource related information obtained from the large language model and retrieve the matching data resources from the vector database formed by the preprocessing module; According to the data lineage relationship of the retrieved data resources and the access time of the data provider, other related extended data resources are associated to form a target data resource set; The actual data of the target data resource set is vectorized and compared with the vector of the required data resource information to find the most similar actual data sample.

2. According to the method for searching data resources on a data sharing open platform based on a large language model in claim 1, it is characterized in that: The embedded model adopts the Chinese-friendly pre-trained model M3E-base.

3. The method for searching data resources on a data sharing open platform based on a large language model according to claim 1, characterized in that: The large language model is ChatGLM, and the prompt is designed in accordance with the CoT thinking chain.

4. The method for searching data resources on a data sharing open platform based on a large language model according to claim 1, characterized in that: The number of data resources that match the search is limited to a maximum of five.

5. The method for searching data resources on a data sharing open platform based on a large language model according to claim 1, characterized in that: The expansion of data resources is achieved by traversing the data lineage network of the retrieved data resources, adding adjacent data resources and other data resources accessed by the same supplier on the same day.

6. The method for searching data resources on a data sharing open platform based on a large language model according to claim 1, characterized in that: The actual number of data samples returned is limited to the five most similar data for each data resource.

7. A data resource search system for a data sharing open platform based on a large language model for implementing the method according to any one of claims 1 to 6, characterized in that: include: The data resource preprocessing module is used to preprocess the full amount of data resources of the data sharing and open platform. It converts the data resource name, classification information, label information, business description information and data fields into vector representation through the embedded model and stores them in the vector database. The data resource retrieval module is used to collect the data requirements of the data user, set prompts and input them into the large language model, analyze and extract the relevant information of the data resources required to solve the requirements, vectorize the relevant information of the data resources obtained from the large language model, and retrieve matching data resources in the vector database formed by the preprocessing module; The extended data resource module is used to associate other related data resources according to the data lineage relationship of the retrieved data resources and the access time of the data supply party to form a target data resource set; The actual data retrieval module performs vectorization processing on the actual data of the target data resource set, compares it with the vector of the required data resource information, and finds the most similar actual data sample.

8. The data resource search system for data sharing and open platform based on large language model according to claim 7 is characterized in that: Also includes: Vector database, used for persistent storage of data resources and vector representation of their actual data.

9. The data resource search system for data sharing and open platform based on large language model according to claim 7, characterized in that: The data resource retrieval module is capable of processing data requirements in natural language form, automatically extracting and matching corresponding data resource features by analyzing the semantic content of the data requirements, and achieving accurate search without the need for the user to have a comprehensive understanding of the data resource list.

10. The data resource search system for data sharing and open platform based on large language model according to claim 7, characterized in that: Before returning the target data resource list and the corresponding data samples, the actual data retrieval module first performs data desensitization processing to ensure that sensitive information is not leaked and that the returned data samples meet the requirements of privacy protection.

Images