A data demand matching method and a data demand pairing method

Abstract

A data demand matching method comprises the following steps: data acquisition: acquiring data uploaded by a data seller, non-normalized description and normalized structured description of the data, and non-normalized description and normalized structured description of the data uploaded by a data buyer; processing the contents uploaded by the data seller and the buyer, wherein word vectors are sorted according to generated confidence values, a word vector description list of the data of the seller and a word vector description list of the data demand of the buyer are generated; and matching degree calculation is performed on the word vector description lists of the seller and the buyer to determine whether the data demands of the buyer and the seller are matched, and a matching result is obtained. A data demand matching method is also provided. Language expression information contained in data is extracted layer by layer, multi-scale feature extraction and fusion are realized at a machine level, the computable advantage of word vectors is utilized, and the efficiency of non-deterministic search and data matching is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology and artificial intelligence, and in particular to a data requirement matching method and a data requirement matching method. BACKGROUND

[0002] Data description refers to the language expression of information such as a certain specific data set, sample or index based on its characteristics. The language expression can be divided into three characteristic levels, including non-normalized description at the natural language level, structured text description conforming to grammar and semantic norms, and machine-level coding and word vector description from top to bottom.

[0003] Data description plays an important role in data transaction and flow process. Data description can enable data operators including humans and trusted AI (artificial intelligence) agents to understand important information of the data without viewing the original data through classification, induction and summary of the data, improve data flow efficiency, and realize privacy protection in data transaction process. In addition, good data description can improve the organization form of data, so that data operators can accurately and efficiently understand data information without sufficient domain knowledge, thereby promoting the matching of data requirements of buyers and sellers.

[0004] In practical applications, data descriptions of different characteristic levels have different applicable scenarios according to their advantages and disadvantages. Non-normalized description at the natural language level can simply and intuitively express the information represented by the data, mainly corresponding to non-professional communication, understanding and induction, and is easy to convert into other forms of requirements, but this description is prone to ambiguity and has high media cost; structured text description conforming to grammar and semantic norms can give clear semantic constraints under natural language, and the word meaning is clear, which can play an important role in professional field expression and data retrieval, but it cannot match different expressions of similar word meanings; machine-level coding and word vector expression can reflect the similar or different characteristics between different samples, which is convenient for further model operation, but it is not friendly to human understanding. However, cross-level data description matching that can be widely applied in different scenarios is difficult to achieve. In addition, due to the existence of complex and diverse features with a wide range of associated spans in the data set, it is difficult for humans to understand, and even domain experts may have incomplete data description problems. The existence of multi-modal data in the data set will further increase the difficulty of data description.

[0005] Various types of databases use structured text descriptions conforming to syntax and semantic specifications as the basic paradigm for data retrieval and matching, which can achieve high efficiency in deterministic retrieval. However, the overly strong semantic constraints greatly reduce the efficiency of non-deterministic search (such as transaction demand matching). To solve this problem, some solutions expand the matching space by adding retrieval keywords and indexes under the existing framework, but this method cannot exhaust all expressions (across the first feature layer and the second feature layer), and has little meaning in practical applications. In addition, with the rapid development of big data technology, the use of structured text expressions alone has not been able to cover the full picture of data description, and the use of multi-modal data such as images and videos for auxiliary description is a hot topic in current scientific research and industrial practice. Purely relying on structured text data description cannot summarize all dimensions of features.

[0006] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0007] The main purpose of the present application is to overcome the defects of the above background art, and provide a data demand matching method and a data demand matching method.

[0008] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0009] A data demand matching method, comprising the following steps:

[0010] (I) Data acquisition:

[0011] Acquiring data uploaded by a data seller, non-normalized description and normalized structured description of the data, and non-normalized description and normalized structured description of the data uploaded by a data buyer;

[0012] (II) Processing of the uploaded content of the data seller:

[0013] Analyzing the data uploaded by the data seller to generate a non-normalized description at the natural language level, and then converting it into a normalized structured description and generating a corresponding confidence value;

[0014] If the data seller uploads a non-normalized description at the natural language level of the data, converting it into a normalized structured description and generating a corresponding confidence value;

[0015] If the data seller uploads a structured text description of the data conforming to syntax and semantic specifications, converting it into a set of word vectors using a text analysis tool;

[0016] For structured text descriptions generated by more than two ways, after converting them into word vectors using text analysis tools, the confidence value is calculated by weighted linear combination;

[0017] According to the generated confidence value, the word vector is sorted, and the word vector description list of the seller data is generated;

[0018] (III) Processing of the uploaded content of the data buyer:

[0019] If the data buyer uploads the natural language level non-standardized description of the data, it is converted into a standardized structured description, and the corresponding confidence value is generated; the generated standardized structured description is converted into a word vector set using text analysis tools;

[0020] If the data buyer uploads a structured text description that meets the grammatical and semantic specifications, it is converted into a word vector set using text analysis tools;

[0021] According to the generated confidence value, the word vector is sorted, and the word vector description list of the buyer's data demand is generated;

[0022] (IV) Judgment of data demand matching:

[0023] Based on the word vector description list of the seller and the buyer, the matching degree is calculated to determine whether the data demand of the buyer and the seller matches, and the matching result is obtained.

[0024] Further:

[0025] Based on the large language model LLM, the data uploaded by the data seller is analyzed to generate natural language level non-standardized description and standardized structured description, and to provide corresponding confidence value; and the natural language level non-standardized description of the data seller or the data buyer is converted into a standardized structured description, and the corresponding confidence value is generated.

[0026] The data uploaded by the data seller includes multi-modal data, and the multi-modal data uploaded by the data seller is analyzed by the large language model LLM to generate natural language level non-standardized description and standardized structured description.

[0027] The judgment of data demand matching includes: based on the cosine similarity, the matching score of data demand is calculated according to the generated data description word vector set, and if the matching score exceeds the threshold, it is determined that the data demand of the buyer and the seller matches.

[0028] The method further includes: if it is determined that it does not match, replacing the data description word vector list of the buyer or the seller, and re-performing data description and demand matching.

[0029] The method further includes: when new seller data or buyer demand enters the system, re-performing data description and demand matching.

[0030] The method is implemented using a data tank system.

[0031] The data vessel system is a data transaction matching system based on blockchain technology to achieve data demand matching.

[0032] A data demand matching method includes: matching the data demands of a seller and a buyer using the data demand matching method, and when a match is successful, performing the data processing and / or communication required for complete data demand matching.

[0033] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data demand matching method or the data demand pairing method.

[0034] The beneficial effects of this invention are:

[0035] The data demand matching method provided by this invention divides the linguistic expressions of data buyers and sellers into three feature levels: non-standardized descriptions at the natural language level, structured text descriptions conforming to grammatical and semantic norms, and word vector descriptions at the machine level. It can utilize a Large Language Model (LLM) to extract the linguistic expression information contained in the data layer by layer, and achieve multi-scale feature extraction and fusion at the machine level. Leveraging the computability of word vectors, it further improves the efficiency of non-deterministic search and data matching. A preferred embodiment utilizes a Large Language Model to support feature extraction and analysis of multimodal data uploaded by users, overcoming the limitation of traditional retrieval methods that cannot simultaneously utilize multimodal data such as images and videos.

[0036] The method of this invention can extract the linguistic information contained in data and achieve multi-scale feature extraction and fusion at the machine level. Leveraging the computability of word vectors, it improves the efficiency of non-deterministic search and data matching. This method can be used for data demand matching in data pools. Attached Figure Description

[0037] Figure 1 This is a flowchart of seller data processing (data description generation) according to an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram of the buyer data processing (data requirement description generation) process according to an embodiment of the present invention.

[0039] Figure 3 This is a flowchart illustrating the data matching process between buyers and sellers according to an embodiment of the present invention. Detailed Implementation

[0040] The embodiments of the present application will be described in detail below. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope of the application or its applications.

[0041] It should be understood that the terms "length", "width", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like, indicate relative or positional relationships based on the orientation or position shown in the drawings, and are used only to facilitate the description of the embodiments of the application and simplify the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the application.

[0042] In addition, the terms "first" and "second" are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features referred to. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0043] The embodiments of the present application provide a data demand matching method, including data acquisition, processing of uploaded content of data sellers and buyers, and judgment of data demand matching, specifically including the following steps:

[0044] Data acquisition: acquiring data uploaded by data sellers, non-normalized description and normalized structured description of data, and non-normalized description and normalized structured description of data uploaded by data buyers.

[0045] Processing of uploaded content of data sellers includes:

[0046] Analyzing the data uploaded by the data seller to generate a non-normalized description at the natural language level, and then converting it into a normalized structured description and generating a corresponding confidence value;

[0047] If the data seller uploads a non-normalized description at the natural language level of the data, it is converted into a normalized structured description and a corresponding confidence value is generated;

[0048] If the data seller uploads a structured text description of the data that conforms to the syntax and semantic specifications, a text analysis tool is used to convert it into a word vector set;

[0049] For structured text descriptions generated by two or more ways, after converting them into word vectors using a text analysis tool, a weighted linear combination is used to calculate the confidence value;

[0050] According to the generated confidence value, the word vectors are sorted to generate a word vector description list of the seller's data.

[0051] For the processing of the uploaded content of the data buyer, including:

[0052] If the data buyer uploads a natural language level non-standardized description of the data, it is converted into a standardized structured description, and a corresponding confidence value is generated; the generated standardized structured description is converted into a word vector set by using a text analysis tool;

[0053] If the data buyer uploads a structured text description conforming to the syntax and semantic specifications, it is converted into a word vector set by using a text analysis tool;

[0054] The word vectors are sorted according to the generated confidence value, and a word vector description list of the data demand of the buyer is generated;

[0055] The judgment of data demand matching includes:

[0056] Based on the word vector description list of the seller and the buyer, the matching degree is calculated, it is judged whether the data demands of the buyer and the seller match, and a matching result is obtained.

[0057] According to the data demand matching method of the embodiment of the application, the language expression of the data buyer and seller is divided into three characteristic levels of natural language level non-standardized description, structured text description conforming to the syntax and semantic specifications, and machine level word vector description. The language expression information contained in the data can be extracted layer by layer by using a large language model (LLM), and multi-scale feature extraction and fusion can be realized at the machine level. The computable advantage of the word vector is utilized to further improve the efficiency of non-deterministic search and data matching. The preferred embodiment utilizes a large language model to support feature extraction and analysis of multi-modal data uploaded by the user, which makes up for the deficiency of traditional retrieval methods that cannot simultaneously utilize multi-modal data such as images and videos.

[0058] The embodiment of the application further provides a data demand matching method, including: using the data demand matching method to match the data demands of the seller and the buyer, and when the matching is successful, performing the data processing and / or communication required for complete data demand matching.

[0059] In some embodiments, the application can be implemented based on a data pot system.

[0060] The specific embodiments of the application are further described below.

[0061] In some specific embodiments, the following steps are included:

[0062] Step B1, the data cauldron system obtains the data uploaded by the data seller and its non-normalized description and standardized structured description. The data can contain multiple modalities, including text, voice, image and video; the data cauldron system obtains the non-normalized description and standardized structured description of the data provided by the data buyer. The data cauldron system judges whether the user is a data seller or a data buyer. If it is a data seller, it enters step B2; if it is a data buyer, it enters step B3.

[0063] Step B2, the data cauldron system analyzes the multi-modal data uploaded by the user using LLM, generates N non-normalized descriptions and M standardized structured descriptions at the natural language level respectively, and provides the corresponding confidence value p data (i,j), representing the confidence value of the jth description of the i-th feature level generated according to the data uploaded by the seller, thereby realizing the extraction of data features at different levels.

[0064] Step B3, judge whether the user has uploaded the natural language level non-normalized description of the data. If uploaded, enter step B31; otherwise, judge whether the user is a data seller or a data buyer. If it is a data seller, it enters step B32; if it is a data buyer, it enters step B4.

[0065] Step B31, the data cauldron system uses LLM to convert the S non-normalized descriptions uploaded by the user into P standardized structured descriptions, where P should be less than S, and provides the corresponding confidence value p denormal (2,m), representing the confidence value of the mth standardized structured description generated according to the non-normalized description. Judge whether the user is a data seller or a data buyer. If it is a data seller, it enters step B32; if it is a data buyer, it uses the pre-trained base model (roberta-base) to convert the generated standardized structured description into a set of word vectors set user_denormal , where the word vectors correspond one-to-one to the generated standardized structured descriptions, and the set contains P word vectors in total. The confidence value of the jth word vector is p user_denornal (j) = p denormal (2,j), enter step B4.

[0066] Step B32, the data cauldron system uses LLM to convert the N non-normalized descriptions generated according to the data uploaded by the seller into Q standardized structured descriptions, where Q should be less than N, and provides the corresponding confidence value p data_denormal (2,j,k), representing the confidence value of the jth standardized structured description generated according to the kth non-normalized description generated in step B2, entering step B4.

[0067] Step B4: Determine whether the user has uploaded a structured text description of the data that conforms to grammatical and semantic standards. If yes, proceed to step B51; otherwise, determine whether the user is a data seller or a data buyer. If the user is a data seller, proceed to step B52; if the user is a data buyer, proceed to step B6.

[0068] Step B51: The data vessel system uses a text analysis tool based on a pre-trained base model (roberta-base) to transform the R normalized descriptions uploaded by the user into a set of word vectors, Set. user_normal This set contains word vectors generated based on the user-input standardized structured description. Each word vector corresponds one-to-one with the user-uploaded standardized structured description, and all word vectors have a confidence value of 1. The system then determines whether the user is a data seller or a data buyer. If the user is a data seller, proceed to step B52; if the user is a data buyer, proceed to step B6.

[0069] Step B52: The data vessel system uses a text analysis tool based on a pre-trained base model (roberta-base) to transform the three types of generated canonical structured descriptions (generated from data, generated from user-input non-canonical descriptions, and generated from data-generated non-canonical descriptions) into a set of word vectors. data_normal Set user_denormal and set data_denormal The confidence value is calculated using a weighted linear combination. (Word vector set) data_normal It contains M word vectors, and the confidence value p of the j-th word vector is... data_normal (j)=p data (2, j). Word vector set user_denormal It contains P word vectors, and the confidence value p of the j-th word vector is... user_denormal (j)=p denormal (2, j). Word vector set data_denormal It contains Q word vectors, and the confidence value of each word vector is calculated using a bilinear weighted method. The confidence value p of the j-th word vector is... data_denormal (j)=p data_denormal (2, j, k)·p data (1, k). Where p data_denormal (2, j, k) represents the confidence value of the j-th normalized structured description generated a second time from the k-th denormalized description generated in step B2, p data (1, k) represents the confidence value of the k-th denormalized description (feature level 1) generated based on the data uploaded by the seller, p data_denormal (j) represents the overall confidence level of the j-th word vector generated from the non-normalized description generated from the data, calculated using bilinear weighting.

[0070] Step B6, the data cauldron system sorts the word vectors according to the confidence value, and generates a word vector description list of the data uploaded by the seller and the data demand of the buyer. For the data buyer, the word vector set set buy =set user_denormal +set user_normal contains P buy +R buy word vectors; for the data seller, the word vector set set sell =set data_normal +set data_denormal +Set user_normal +Set user_denormal contains M+P sell +Q+R sell word vectors.

[0071] Step B7, matching of data demand of the buyer and the seller. Based on the computable advantage of the word vector, the matching score of the data demand is calculated based on the cosine similarity according to the generated data description word vector set, and then it is judged whether the data demand of the buyer and the seller is matched. The calculation formula is as follows: Wherein, sim[set buy (k), set sell (l)] represents the cosine similarity of the kth word vector in the word vector set of the buyer and the lth word vector in the word vector set of the seller, that is If the matching score of the data demand exceeds the threshold value K, it is determined that the transaction demand of the buyer and the seller is matched, and step B8 is entered; if the matching score of the data demand does not exceed the threshold value K, the data description word vector list of the buyer or the seller is replaced, and the data description and demand matching is performed again, and B1 is entered.

[0072] Step B8, when the new seller data or the buyer demand enters the data cauldron system, the data description and demand matching is performed again, and B1 is entered.

[0073] Through the method of the application, the language expression information contained in the data can be extracted layer by layer based on the LLM, and the multi-scale feature extraction and fusion is realized at the machine level, the computable advantage of the word vector is utilized, and the efficiency of the non-deterministic search and data matching is improved.

[0074] The application embodiment further provides a storage medium for storing a computer program, which is executed to perform at least the method described above.

[0075] The application embodiment further provides a control device, which comprises a processor and a storage medium for storing a computer program; wherein the processor is used to execute the computer program to perform at least the method described above.

[0076] The embodiments of the present application also provide a processor, which executes a computer program and at least performs the method described above.

[0077] The storage medium can be implemented by any type of volatile or nonvolatile storage devices, or a combination thereof. The storage medium described in the embodiments of the present application is intended to include, but not limited to, these and any other suitable types of memory.

[0078] In several embodiments of the present application, it should be understood that the disclosed system and method can be implemented in other manners. The described device embodiments are merely schematic, and the division of the units is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling, or direct coupling or communication connection between the components can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0079] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0080] In addition, each functional unit in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate physical unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in the form of hardware, or in the form of hardware plus software function units.

[0081] Those skilled in the art can understand that all or part of the steps of the above method embodiments can be completed by relevant hardware instructed by programs. The foregoing programs can be stored in a computer readable storage medium, and when the programs are executed, the steps of the above method embodiments are executed. The foregoing storage medium includes mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, and various media that can store program codes.

[0082] Alternatively, the above-mentioned integrated unit of the present application, if realized in the form of a software function module and sold or used as an independent product, can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present application. The aforementioned storage medium includes: mobile storage devices, ROM, RAM, magnetic disks or optical disks, and various other media that can store program codes.

[0083] The methods disclosed in the several method embodiments of the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0084] The features disclosed in the several product embodiments of the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0085] The features disclosed in the several method or device embodiments of the present application can be combined arbitrarily without conflict to obtain new method or device embodiments.

[0086] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of equivalent substitutions or obvious modifications can be made, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present application.

Claims

1. A data demand matching method, characterized by, Comprising the following steps: (I) Data acquisition: Acquiring data uploaded by data sellers, non-normalized descriptions and normalized structured descriptions of data, and non-normalized descriptions and normalized structured descriptions of data uploaded by data buyers; (II) Processing of the uploaded content by the data seller: Analyzing the data uploaded by the data seller, generating a non-normalized description at the natural language level, and then converting it into a normalized structured description, and generating a confidence value representing the confidence of this normalized structured description; If the data seller uploads a non-normalized description at the natural language level of the data, convert it into a normalized structured description, and generate a confidence value representing the confidence of this normalized structured description; If the data seller uploads a structured text description of the data that conforms to the syntax and semantic specifications, use text analysis tools to convert it into a set of word vectors; For structured text descriptions generated by more than two methods, after converting them into word vectors using text analysis tools, use weighted linear combination to calculate the confidence value for subsequent word vector sorting and data matching degree calculation; Sort the word vectors according to the generated confidence values to generate a word vector description list of the seller's data; (III) Processing of the uploaded content by the data buyer: If the data buyer uploads a non-normalized description at the natural language level of the data, convert it into a normalized structured description, and generate a confidence value representing the confidence of this normalized structured description; use text analysis tools to convert the generated normalized structured description into a set of word vectors; If the data buyer uploads a structured text description that conforms to the syntax and semantic specifications, use text analysis tools to convert it into a set of word vectors; Sort the word vectors according to the generated confidence values to generate a word vector description list of the buyer's data requirements; (IV) Judgment of data requirement matching: Based on the word vector description lists of the seller and the buyer, calculate the matching degree to determine whether the data requirements of the buyer and the seller match, and obtain the matching result.

2. The data requirement matching method of claim 1, wherein, Based on a large language model LLM, analyze the data uploaded by the data seller to generate a non-normalized description at the natural language level and a normalized structured description, and provide the corresponding confidence value; and convert the non-normalized description at the natural language level of the data seller or the data buyer into a normalized structured description, and generate the corresponding confidence value.

3. The data requirement matching method of claim 2, wherein, The data uploaded by the data seller includes multi-modal data, and the large language model LLM is used to analyze the multi-modal data uploaded by the data seller to generate a non-normalized description at the natural language level and a normalized structured description.

4. The data demand matching method of any one of claims 1 to 3, wherein, The judgment of data requirement matching includes: based on the cosine similarity, calculating the matching score of the data requirements according to the generated data description word vector set, and if the matching score exceeds the threshold, determining that the data requirements of the buyer and the seller match.

5. The data demand matching method of any one of claims 1 to 3, wherein, Further comprising: If it is determined that it does not match, replace the data description word vector list of the buyer or the seller and re-perform data description and requirement matching.

6. The data demand matching method of any one of claims 1 to 3, wherein, Further comprising: When new seller data or buyer requirements enter the system, re-perform data description and requirement matching.

7. The data demand matching method of any one of claims 1 to 3, wherein, Implemented using a data pot system.

8. The data requirement matching method of claim 7, wherein, The data vessel system is a data transaction matching system based on blockchain technology to achieve data demand matching.

9. A data demand matchmaking method, characterized in that, include: The data demand matching method as described in any one of claims 1 to 8 is used to match the data demands of the seller and the buyer. When a match is successful, the data processing and / or communication required for complete data demand matching are performed.

10. A computer readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the data demand matching method as described in any one of claims 1 to 8, or the data demand matching method as described in claim 9.

Images