Artificial intelligence corpus useful proof of work data generation method and system

Abstract

The application provides an AI corpus useful work proof data generation method and system, which comprises the following steps: obtaining AI corpus data to be processed and classifying; knowledge extraction is performed on each type of corpus data to generate metadata; difficulty coefficients are determined according to metadata types and weighted calculation of comprehensive difficulty coefficients; time cost, computing power cost and Token cost in the node processing process are obtained; the useful work proof cost is calculated based on the comprehensive difficulty coefficient and the cost parameter; the computing power proof data set containing the source corpus identifier, the node ID, the timestamp and the PoUW cost is generated; and the record containing the structured corpus, the hash value, the timestamp, the data set and the private key is returned. The application quantifies the computing power consumption by multi-dimensional difficulty weighting and cost integration, solves the problem of no actual value of the traditional work proof, and realizes the right protection and compliance audit of the AI corpus in the decentralized environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of AI corpus processing, in particular to an AI corpus useful proof of work data generation method and system. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, high-quality AI corpus (including text, speech, image, etc.) has become the basis for training and optimizing AI models. AI corpus needs to go through cleaning, labeling, and formatting processing steps to extract structured knowledge and information from unstructured data, which is called knowledge extraction. Knowledge extraction involves a large amount of computing resource consumption, including computing power, storage, and bandwidth, etc.

[0003] In the field of blockchain, proof of work (PoW) is a common consensus mechanism that ensures network security by requiring nodes to complete hash operations that are difficult to calculate but easy to verify. However, traditional PoW calculation itself does not produce actual application value, resulting in waste of computing power resources. Therefore, proof of useful work (PoUW) is proposed, which aims to use computing power to solve practical problems (such as scientific computing, AI training), thereby maintaining network security while generating additional value. However, the current implementation of PoUW mainly focuses on replacing the calculation task with useful calculation, and has not yet systematically recorded and verified the computing power consumed during the calculation process, which cannot provide verifiable computing power proof for subsequent data auditing, rights confirmation, etc.

[0004] In the AI corpus trading scenario, data rights confirmation and value measurement are key issues. Existing technologies usually rely on centralized platforms for data auditing, but it is difficult to meet the trusted verification needs in a decentralized environment. For example, public patent application CN202111212911.7 proposes a host intrusion detection method that uses useful proof of work for security verification, but does not involve converting computing power consumption into data products. Another public patent application CN202011546640.4 analyzes the performance indicators of PoUW, providing guidance for consensus mechanism optimization, but still does not solve the problem of how to form a verifiable data set from the computing power consumption process itself.

[0005] Therefore, a technical solution is needed that can structure and verify the computing power consumed in the AI corpus knowledge extraction process, generate auditable computing power proof data sets, and support the rights confirmation and transaction compliance requirements of AI corpus in a decentralized environment. SUMMARY

[0006] In view of the defects in the prior art, the purpose of the present application is to provide an AI corpus useful proof of work data generation method and system.

[0007] A method for generating useful proof-of-work data for AI corpora according to the present invention includes: Acquire the AI ​​corpus data to be processed; The AI ​​corpus data is classified to obtain corpus data of multiple categories; Knowledge extraction is performed on the corpus data for each category to generate corresponding metadata; The difficulty coefficient is determined based on the type of the metadata, and the difficulty coefficients of each category are weighted and calculated to obtain the comprehensive difficulty coefficient. The cost parameters of the node processing the corpus data are obtained, and the cost of useful work proof is calculated based on the comprehensive difficulty coefficient and cost parameters. Based on the useful proof-of-work cost, and combined with the source corpus data, processing time, node identifier, and timestamp, a proof-of-computation dataset is generated.

[0008] Preferably, the metadata includes one or more of the following: structured documents, unstructured documents, semi-structured documents, audio, video, images, formulas, or tables.

[0009] Preferably, the step of determining the difficulty coefficient based on the type of the metadata and performing a weighted calculation of the difficulty coefficients for each category to obtain a comprehensive difficulty coefficient further includes: Different difficulty levels are assigned to document-type metadata based on their degree of structure. Assign audio / video difficulty coefficients to audio / video metadata; Assign corresponding difficulty levels to metadata such as images, formulas, and tables; The overall difficulty coefficient is obtained by integrating or summing the various difficulty coefficients.

[0010] Preferably, the cost parameters include one or more of the following: time cost, computing power cost, token cost, server computing power, server processing time, power consumption, large model input token amount, and large model output token amount.

[0011] Preferably, in the step of calculating the cost of useful work proof based on the comprehensive difficulty coefficient and cost parameters, the following formula is used for calculation:

[0012] in, To assess the overall difficulty level, To reduce the processing cost of large models, This refers to the cost of server computing power.

[0013] Preferably, the computational power proof dataset includes the following fields: source corpus dataset identifier, node identifier, timestamp, and useful work proof cost.

[0014] Preferably, the node identifier is a node ID or a confirmation number, and the timestamp is the recording time when the processing is completed.

[0015] A system for generating useful proof-of-work data for AI corpora according to the present invention includes: The acquisition module is used to acquire AI corpus data to be processed; The classification module is used to classify the AI ​​corpus data to obtain corpus data of multiple categories; The extraction module is used to extract knowledge from the corpus data of each category and generate corresponding metadata; The difficulty calculation module is used to determine the difficulty coefficient based on the type of the metadata, and to perform a weighted calculation on the difficulty coefficients of each category to obtain a comprehensive difficulty coefficient; The cost acquisition module is used to acquire cost parameters during the node's processing of the corpus data. The cost parameters include at least one of time cost, computing power cost, and token cost. The cost calculation module is used to calculate the cost of useful work proof based on the comprehensive difficulty coefficient and cost parameters. The dataset generation module is used to generate a computing power proof dataset based on the useful work proof cost, combined with the source corpus data, processing time, node identifier and timestamp; The return module is used to return a record containing structured corpus data, dehash values, timestamps, proof-of-computation datasets, and private keys.

[0016] Preferably, the difficulty calculation module further includes: The document difficulty submodule is used to assign different difficulty levels to document class metadata according to its degree of structure. The audio / video difficulty submodule is used to assign audio / video difficulty coefficients to audio / video class metadata; Other difficulty sub-modules are used to assign corresponding difficulty levels to image, formula, and table metadata respectively; The comprehensive calculation submodule is used to perform integration or summation operations on various difficulty coefficients to obtain the comprehensive difficulty coefficient.

[0017] According to the present invention, a computer-readable storage medium is provided thereon storing a computer program, which, when executed by a processor, implements the steps of the AI ​​corpus useful work proof data generation method.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention classifies AI corpora in multiple dimensions and assigns weighted difficulty coefficients to each. It then performs integral calculations based on time costs, computing power costs, and token costs in the node processing process. This achieves the technical effect of quantifying the computing power consumption of the originally abstract corpus knowledge extraction into a standardized and auditable "useful proof-of-work cost". This solves the problem that traditional PoW computation has no practical application value and realizes a reliable mapping between computing power resource consumption and data value generation.

[0019] 2. This invention generates a structured computational power proof dataset by packaging the source corpus identifier, node ID, timestamp, and the calculated useful work proof cost, and uses private key signing to return records containing dehash values, thereby achieving the technical effect of providing tamper-proof computational power consumption proof for AI corpora in a decentralized environment.

[0020] 3. This invention introduces the ratio of the input / output token cost of a large model to the server's computing power and processing time as a computing power cost factor, and uses an integral formula for comprehensive calculation, thereby achieving the technical effect of differentiated and accurate billing for different types of large models, different hardware configurations, and different corpus difficulties. Attached Figure Description

[0021] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the PoUW data generation process according to an embodiment of the present invention. Detailed Implementation

[0022] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0023] Example 1 This embodiment provides a method for generating useful work proof data for AI corpora, such as... Figure 1 As shown in the figure, the entire process from raw corpus input, through classification, knowledge extraction, difficulty weighting, cost parameter collection, PoUW cost calculation, to finally outputting a computational power proof dataset and records. This method includes the following steps: Step S1: Obtain the AI ​​corpus data to be processed.

[0024] In this embodiment, the AI ​​corpus data can be a collection of various unstructured or semi-structured data, such as text, speech, images, and videos. Examples include documents crawled from the internet, publicly available speech datasets, and image annotation sets. The system obtains the original corpus dataset, denoted as source_corpus_dataset, through a data interface or distributed storage.

[0025] Step S2: Classify the AI ​​corpus data to obtain corpus data of multiple categories.

[0026] The system divides the original corpus into several subcorpora according to data type, denoted as Ci (i=1,2,…,n), with each subcorpus representing a logically independent subset of the corpus. For example, it can be divided into text subcorpora, audio subcorpora, video subcorpora, image subcorpora, etc., according to modality, or it can be further subdivided according to domain or source.

[0027] Step S3: Extract knowledge from the corpus data of each category and generate corresponding metadata.

[0028] For each subcorpus Ci, the system performs knowledge extraction on each data unit within it. Specifically, the following method is used: For text-based data, each document (Di) is categorized into structured documents (Ds), unstructured documents (Dns), or semi-structured documents (Dhs) based on its internal structure. Structured documents are those with clearly defined, fixed fields whose content can be accessed directly by field names, such as JSON format data or documents defined by XML Schema. Unstructured documents are free text without a predefined data model or field structure, such as news reports, social media posts, and emails; knowledge extraction from these documents is the most challenging. Semi-structured documents fall between structured and unstructured documents.

[0029] For audio data (Au) and video data (Vi), extract metadata such as speech features, audio tracks, and keyframes.

[0030] For image data (Fi), extract object labels, bounding boxes, segmentation masks, etc.

[0031] For formulas (Fo) and tables (Ta), extract their semantic structure and content.

[0032] The extraction process can employ pre-trained large language models, convolutional neural networks, or traditional natural language processing and image processing tools. Metadata is stored in key-value pairs or JSON format.

[0033] Step S4: Determine the difficulty coefficient based on the type of the metadata, and perform a weighted calculation on the difficulty coefficients of each category to obtain the comprehensive difficulty coefficient; This embodiment provides a method for calculating the difficulty coefficient. For document-type metadata, the difficulty coefficient D(Ds,Dns,Dhs) is set as follows: unstructured documents have the highest difficulty, set to 0.8; semi-structured documents are next, set to 0.5; and structured documents have the lowest difficulty, set to 0.2. For audio and video-type metadata, the difficulty coefficient depends on the content complexity, such as audio clarity and video frame rate, and ranges from 0.3 to 0.9. For images Fi, formulas Fo, and tables Ta, fixed difficulty coefficients are assigned: images 0.6, formulas 0.9, and tables 0.7. In a preferred embodiment, the difficulty coefficients of the above metadata can be adjusted according to the specific data source and requirements.

[0034] The overall difficulty coefficient S is calculated by integration (summation):

[0035] In practice, the difficulty coefficients of each type of metadata can be weighted and accumulated. For example, if a subcorpus contains 10 unstructured documents, 5 minutes of audio, and 2 images, then the overall difficulty coefficient S = 10 × 0.8 + 5 × 0.6 + 2 × 0.6 = 12.2.

[0036] Step S5: Obtain cost parameters during the node's processing of the corpus data. The cost parameters include at least one of time cost, computing power cost, and token cost.

[0037] In this embodiment, the node participating in the processing is a computing node in a distributed computing network. The node configuration parameters include: Server computing power capability Se: Different computing power factors are assigned based on the server type, for example, Se=1.5 for GPU servers and Se=0.8 for CPU servers; Server processing time Ts: The clock time (in seconds) taken by a node to process this sub-corpus; Electricity consumption Es: Optional, for subsequent carbon emissions audits; Large model input token count Ti: The number of tokens input when calling the large language model; To: The number of tokens output by the model.

[0038] The weighted cost of tokens is calculated as: Token = Ti × To. Then, based on the large model type LLM used, the cost per token is determined, and the large model processing cost LLMc = Token × unit price is calculated. For example, if a large model charges $0.01 for every 1000 tokens input and $0.03 for every 1000 tokens output, the actual cost is calculated linearly based on Ti and To.

[0039] Server computing cost Sc is defined as the ratio of processing time to computing power: Sc = Ts / Se. This value reflects the relative computing resources consumed by a node to process the corpus.

[0040] Step S6: Calculate the cost of useful work proof based on the comprehensive difficulty coefficient and cost parameters.

[0041] This embodiment uses the following formula to calculate the useful proof-of-work cost of a single subcorpus Ci:

[0042] in The overall difficulty coefficient obtained in step S4. To reduce the processing cost of large models, This represents the server's computing power cost. The total cost of proof-of-work for the entire source corpus dataset is the sum of the costs of each subcorpus:

[0043] For example, a certain subcorpus =12.2, =0.5 USD, =100 seconds / 1.5 ≈ 66.7, then ≈12.2×0.5×66.7≈406.87 (The unit can be defined as "computing power points" or "CoinHour").

[0044] Step S7: Based on the useful proof-of-work cost, and combined with the source corpus data, processing time, node identifier, and timestamp, generate a computational power proof dataset.

[0045] The system packages the results of the entire processing into a structured computing power proof dataset (DataSet), which includes the following fields: source_corpus_dataset: A unique identifier or hash value for the original corpus set; NodeId: The ID or authorization number of the processing node; timestamp: The Unix timestamp when processing is complete; pouw_cost: The total cost of proving useful work calculated in step S6.

[0046] In a preferred embodiment, the system can be further expanded to include detailed metadata, difficulty factor decomposition, cost decomposition, etc., for each subcorpus, for subsequent auditing.

[0047] Step S8: Return a record containing structured corpus data, dehash value, timestamp, computational power proof dataset, and private key.

[0048] The processing node uses its private key to digitally sign the computing power proof dataset, generates a dehash value, and ultimately returns a complete record, including: the structured corpus data; the hash value of the dataset; a timestamp; the computing power proof dataset; and the node's private key signature or public key. This record can be submitted to a blockchain network or other distributed ledgers for subsequent computing power verification, data ownership confirmation, and transaction compliance auditing.

[0049] Example 2 This embodiment also provides a system for generating useful proof-of-work data for AI corpora, used to execute the above methods. The system includes the following modules: Acquisition module: Used to acquire AI corpus data to be processed, supporting data reading from local file system, IPFS, HDFS and other storage.

[0050] Classification module: Used to classify corpus data according to file extension, MIME type or content features, and output multiple subcorpora.

[0051] Extraction module: Internally integrates a text analyzer, audio processor, image recognition model, etc., to extract knowledge from each sub-corpus and generate metadata.

[0052] Difficulty Calculation Module: Query the preset difficulty coefficient table based on the metadata type, and perform weighted summation or integral operation to output the comprehensive difficulty coefficient.

[0053] Cost Acquisition Module: Collects system metrics (processing time, CPU / GPU utilization, token counter) and configuration parameters (server type, unit price of large models) during node operation.

[0054] Cost calculation module: Calculate the PoUW cost according to the formula in step S6.

[0055] Dataset generation module: Assembles the original corpus identifiers, node IDs, timestamps, and PoUW costs into a structured dataset.

[0056] Return module: performs a hash signature on the dataset and returns a record containing the structured corpus, hash value, timestamp, dataset, and signature.

[0057] The modules communicate with each other via message queues or API interfaces. The system can be deployed on a single server or run on distributed computing nodes.

[0058] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0059] For example, the difficulty calculation module can be designed as a dedicated arithmetic logic unit in an FPGA to calculate the difficulty coefficients of multiple sub-corpora in parallel. The cost calculation module can perform multiplication and addition operations quickly using an embedded DSP chip. Such hardware implementations also fall within the scope of protection of this invention.

[0060] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for generating useful work proof data for AI corpora, characterized in that, include: Acquire the AI ​​corpus data to be processed; The AI ​​corpus data is classified to obtain corpus data of multiple categories; Knowledge extraction is performed on the corpus data for each category to generate corresponding metadata; The difficulty coefficient is determined based on the type of the metadata, and the difficulty coefficients of each category are weighted and calculated to obtain the comprehensive difficulty coefficient. The cost parameters of the node processing the corpus data are obtained, and the cost of useful work proof is calculated based on the comprehensive difficulty coefficient and cost parameters. Based on the useful proof-of-work cost, and combined with the source corpus data, processing time, node identifier, and timestamp, a proof-of-computation dataset is generated.

2. The method for generating useful work proof data for AI corpora according to claim 1, characterized in that, The metadata includes one or more of the following: structured documents, unstructured documents, semi-structured documents, audio, video, images, formulas, or tables.

3. The method for generating useful work proof data for AI corpora according to claim 1, characterized in that, The step of determining the difficulty coefficient based on the type of the metadata and performing a weighted calculation of the difficulty coefficients for each category to obtain a comprehensive difficulty coefficient further includes: Different difficulty levels are assigned to document-type metadata based on their degree of structure. Assign audio / video difficulty coefficients to audio / video metadata; Assign corresponding difficulty levels to metadata such as images, formulas, and tables; The overall difficulty coefficient is obtained by integrating or summing the various difficulty coefficients.

4. The method for generating useful work proof data for AI corpora according to claim 1, characterized in that, The cost parameters include one or more of the following: time cost, computing power cost, token cost, server computing power, server processing time, power consumption, input token amount of large models, and output token amount of large models.

5. The method for generating useful work proof data for AI corpora according to claim 1, characterized in that, In the step of calculating the cost of useful work proof based on the comprehensive difficulty coefficient and cost parameters, the following formula is used for calculation: in, To assess the overall difficulty level, To reduce the processing cost of large models, This refers to the cost of server computing power.

6. The method for generating useful work proof data for AI corpora according to claim 1, characterized in that, The computing power proof dataset includes the following fields: Source corpus dataset identifier, node identifier, timestamp, and cost of useful work proof.

7. The method for generating useful work proof data for AI corpora according to claim 1, characterized in that, The node identifier is a node ID or a confirmation number, and the timestamp is the recording time when the processing is completed.

8. A system for generating useful work proof data for AI corpora, characterized in that, include: The acquisition module is used to acquire AI corpus data to be processed; The classification module is used to classify the AI ​​corpus data to obtain corpus data of multiple categories; The extraction module is used to extract knowledge from the corpus data of each category and generate corresponding metadata; The difficulty calculation module is used to determine the difficulty coefficient based on the type of the metadata, and to perform a weighted calculation on the difficulty coefficients of each category to obtain a comprehensive difficulty coefficient; The cost acquisition module is used to acquire cost parameters during the node's processing of the corpus data. The cost parameters include at least one of time cost, computing power cost, and token cost. The cost calculation module is used to calculate the cost of useful work proof based on the comprehensive difficulty coefficient and cost parameters. The dataset generation module is used to generate a computing power proof dataset based on the useful work proof cost, combined with the source corpus data, processing time, node identifier and timestamp; The return module is used to return a record containing structured corpus data, dehash values, timestamps, proof-of-computation datasets, and private keys.

9. The AI ​​corpus useful work proof data generation system according to claim 8, characterized in that, The difficulty calculation module further includes: The document difficulty submodule is used to assign different difficulty levels to document class metadata according to its degree of structure. The audio / video difficulty submodule is used to assign audio / video difficulty coefficients to audio / video class metadata; Other difficulty sub-modules are used to assign corresponding difficulty levels to image, formula, and table metadata respectively; The comprehensive calculation submodule is used to perform integration or summation operations on various difficulty coefficients to obtain the comprehensive difficulty coefficient.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the AI ​​corpus useful work proof data generation method as described in any one of claims 1 to 7.

Images