Data collecting and processing method and system

Abstract

The invention relates to the field of data processing, and provides a data collecting and processing method and system in order to improve the processing efficiency of mass data, a data recognition model is created to automatically process data to improve the processing efficiency, and a recognition result is manually rechecked to improve the recognition accuracy. During data processing, the sequence of recognition and the review scale of manual review are determined according to the comprehensive credibility of the data collection nodes, the reliability of a data provider is reflected by the comprehensive credibility, and data collected by the data collection nodes with high comprehensive credibility are preferentially recognized so as to quickly extract data with higher reference value; the manual rechecking scale of the data collected by the data collection node with low comprehensive credibility is stricter so as to ensure the data accuracy.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular, to a data collection and processing method and system. Background Art

[0002] In the process of cracking down on the production and sales of fake and shoddy products, multiple processes are involved, such as: intelligence collection, intelligence identification, case investigation, terminal visits, arrest level, recommended rights protection, initiation of rights protection, preparation of litigation materials, case authorization, case registration, registration of trial results, execution of trial results, and settlement of attorney fees. In the process of intelligence collection and intelligence identification, how to improve the processing efficiency of massive data has become an urgent problem to be solved. Summary of the Invention

[0003] In order to improve the processing efficiency of massive data, the present application provides a data collection and processing method and system.

[0004] The technical solution adopted by the present invention to solve the above problems is:

[0005] A data collection and processing method, including:

[0006] Step 1: Obtain the comprehensive trustworthiness of the data collection node;

[0007] Step 2: Create a data identification model to identify the authenticity of data;

[0008] Step 3: Based on the data identification model, identify the data collected by all data collection nodes and conduct manual review on the identification results. For the data collected by data collection nodes with high comprehensive trustworthiness, priority is given to identification. For the data collected by data collection nodes with low comprehensive trustworthiness, the manual review criteria are more stringent.

[0009] Further, the calculation method of the comprehensive trustworthiness is:

[0010] Step 11: Obtain the direct trustworthiness of the current data collection node and the recommended trustworthiness of other data collection nodes associated with the current data collection node for the current data collection node;

[0011] Step 12: Calculate the comprehensive trustworthiness of the current data collection node according to the direct trustworthiness and the recommended trustworthiness.

[0012] Further, the direct trustworthiness is determined according to the historical information of the current data collection node.

[0013] Further, the calculation method of the recommended trustworthiness is: determine the weights of each associated node according to the similarity with the current data collection node; calculate the recommended trustworthiness according to the weights of each associated node and the direct trustworthiness of each associated node.

[0014] Further, the comprehensive trustworthiness Ttotal The calculation formula of T is: total = w d * T d + w re * T re where T d is the direct trust degree, T re is the recommended trust degree, w d and w re are the weights of the direct trust degree and the recommended trust degree respectively, and w d + w re = 1.

[0015] Furthermore, the data recognition model includes recognition units created by multiple independent artificial intelligence algorithms, and the final output result of the data recognition model is the comprehensive value of the output results of each recognition unit.

[0016] Furthermore, it also includes re-obtaining the comprehensive trust degree of the data collection node according to the trigger condition.

[0017] Furthermore, the trigger condition is time or event.

[0018] The data collection and processing system includes:

[0019] The comprehensive trust degree acquisition module: used to acquire the comprehensive trust degree of the data collection node;

[0020] The model creation module: used to create a data recognition model;

[0021] The data processing module: based on the data recognition model, identifies the data collected by all data collection nodes and conducts manual review on the recognition results. For the data collected by the data collection nodes with high comprehensive trust degree, the identification is given priority, and for the data collected by the data collection nodes with low comprehensive trust degree, the manual review criteria are more strict.

[0022] Furthermore, the comprehensive trust degree acquisition module includes:

[0023] The direct trust degree acquisition unit: used to acquire the direct trust degree of the current data collection node;

[0024] The recommended trust degree acquisition unit: used to acquire the recommended trust degree of other data collection nodes associated with the current data collection node for the current data collection node;

[0025] The comprehensive trust degree calculation unit: used to calculate the comprehensive trust degree of the current data collection node according to the direct trust degree and the recommended trust degree.

[0026] The beneficial effects of the present invention compared with the prior art are as follows: creating a data recognition model to automate the processing of data to improve processing efficiency, and performing manual review on the recognition results to improve recognition accuracy; when processing data, determining the order of recognition and the review scale of manual review according to the comprehensive trustworthiness of data collection nodes. The comprehensive trustworthiness reflects the reliability of data providers. For the data collected by data collection nodes with high comprehensive trustworthiness, priority recognition is carried out to quickly extract more valuable reference data, and for the data collected by data collection nodes with low comprehensive trustworthiness, the manual review scale is more stringent to ensure data accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flowchart of a data collection and processing method;

[0028] Figure 2 is an architecture diagram of a data collection and processing system. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0030] As Figure 1 shown, a data collection and processing method includes:

[0031] Step 1: Obtain the comprehensive trustworthiness of data collection nodes.

[0032] Taking the case of anti-counterfeiting and rights protection as an example, data collection nodes can be participants such as suppliers, manufacturers, distributors, etc. These nodes play different roles in the supply chain and sales network, and their behaviors and reputations directly affect product quality and market order. By calculating the comprehensive trustworthiness, the reliability of the information provided by each node can be evaluated. In addition, it can also be determined which nodes have a higher probability of engaging in counterfeiting and shoddy behaviors based on the comprehensive trustworthiness, so as to conduct targeted investigations and treatments.

[0033] The comprehensive trustworthiness can be determined according to a single parameter such as direct trustworthiness or referral trustworthiness.

[0034] The direct trustworthiness is determined based on the node's historical performance and service quality and other self-information. Taking a supplier as an example, its direct trustworthiness can be evaluated by analyzing indicators such as the quality of products supplied in the past, delivery punctuality, and after-sales service satisfaction. When calculating specifically, methods such as weighted average can be used to assign corresponding weights to each indicator according to its importance, and then the direct trustworthiness value can be obtained comprehensively.

[0035] The calculation of recommended trust needs to consider factors such as the similarity between the recommended node (a node having a certain association with the target node, such as business dealings, jointly participating in a transaction, etc.) and the target node, as well as the trust of the recommended node itself.

[0036] When calculating the recommended trust, first identify other nodes that are associated with the target node. For example, in a supply chain, other suppliers with similar product types, industry backgrounds, or transaction records as the target supplier can be used as recommended nodes.

[0037] Then calculate the similarity between the recommended node and the target node. Multiple methods can be used, such as similarity based on product features, similarity based on transaction behavior, similarity based on geographical location, etc. For example, the similarity based on product features can be calculated by comparing the main components, functional parameters, etc. of the two products.

[0038] Next, determine the recommendation weight of the recommended node according to the level of similarity. The higher the similarity of the node, the greater its recommendation weight, and the greater the impact on the trustworthiness evaluation of the target node. For example, the weight allocation can adopt the linear normalization method, map the similarity value to the interval [0, 1], and ensure that the sum of the weights of all recommended nodes is 1. For example, assume there are three recommended nodes A, B, and C, and their similarities with the target node are 0.8, 0.6, and 0.4 respectively. After normalization, their recommended weights can be determined as 0.4, 0.3, and 0.2 respectively. In this way, in the subsequent calculation of recommended trust, the opinion of recommended node A will account for the largest proportion.

[0039] Finally, multiply the trust value (i.e., the comprehensive trust value) of each recommended node by its corresponding recommendation weight, and then sum the results of all recommended nodes to obtain the recommended trust value of the target node. For example, if the trust values of recommended nodes A, B, and C are 0.9, 0.7, and 0.6 respectively, and the corresponding weights are 0.4, 0.3, and 0.2, then the recommended trust value of the target node is calculated as: 0.9×0.4 + 0.7×0.3 + 0.6×0.2 = 0.36 + 0.21 + 0.12 = 0.69. This indicates that according to the evaluation of the recommended nodes, the recommended trust value of the target node is 0.69. The comprehensive trust value of the initial node can directly adopt its direct trust value.

[0040] Furthermore, when calculating the recommended trustworthiness, the evaluation difference degree of the recommended node with respect to the target node can also be considered. The evaluation difference degree of the recommended node with respect to the target node is used to measure the difference between the trustworthiness evaluations of different recommended nodes for the same target node. This difference degree can help evaluate whether the evaluations of the recommended nodes are consistent and whether there may be biases or anomalies in the evaluations of certain recommended nodes. For example, a low difference degree indicates that the evaluation of the current recommended node is relatively consistent with the evaluations of other recommended nodes, which shows that the evaluation of this recommended node is relatively reliable; on the contrary, it indicates that there is a large deviation between the evaluation of the recommended node and the evaluations of other recommended nodes, which may indicate that there are biases or anomalies in the evaluation of this recommended node. When calculating the recommended trustworthiness value, the weight of the recommended node can be adjusted according to the evaluation difference degree. For recommended nodes with a large evaluation difference degree, their weights can be appropriately reduced. Since the similarity is different, the trustworthiness evaluations may vary greatly. Therefore, the associated nodes can be classified according to the similarity, and the trustworthiness evaluations can be compared among the nodes with the same type of similarity.

[0041] Using a single parameter to calculate the comprehensive trustworthiness is not comprehensive enough. In this embodiment, the comprehensive trustworthiness of the current data collection node is calculated based on the direct trustworthiness and the recommended trustworthiness. The comprehensive trustworthiness T total The calculation formula is: T total = w d * T d + w re * T re , where, T d is the direct trustworthiness, T re is the recommended trustworthiness, w d and w re are the weights of the direct trustworthiness and the recommended trustworthiness respectively, and w d + w re = 1.

[0042] The comprehensive trustworthiness is not static and needs to be dynamically updated according to its behavior and performance. The trustworthiness value of the node can be adjusted in a timely manner through the time-triggered and event-triggered mechanisms to reflect its latest trustworthy state.

[0043] Step 2: Create a data identification model to identify the authenticity of the data.

[0044] Using a single artificial intelligence algorithm to create a data recognition model is prone to bias and misjudgment. Therefore, in this embodiment, multiple independent artificial intelligence algorithms are used to create a model, and the characteristics of counterfeit and shoddy products, such as product pictures, description texts, price anomalies, etc., are identified by training the model. Each algorithm runs independently to generate its own output results, and then a voting mechanism is introduced to integrate the output results of multiple algorithms. The voting mechanism is based on weight allocation, majority principle, or other appropriate rules. For example, different weights are assigned to algorithms according to factors such as their historical performance and accuracy, and the final identification conclusion is determined based on the weighted voting results. This mechanism not only improves the accuracy of identification but also enhances the robustness of the system.

[0045] Step 3: Based on the data recognition model, identify the data collected by all data collection nodes and conduct manual review on the identification results. For the data collected by data collection nodes with high comprehensive trust, priority is given to identification, and for the data collected by data collection nodes with low comprehensive trust, the manual review criteria are more stringent.

[0046] Correspondingly, this embodiment also provides a data collection and processing system, as Figure 2 shown, including:

[0047] Comprehensive trust degree acquisition module: used to acquire the comprehensive trust degree of data collection nodes;

[0048] Model creation module: used to create a data recognition model;

[0049] Data processing module: Based on the data recognition model, identify the data collected by all data collection nodes and conduct manual review on the identification results. For the data collected by data collection nodes with high comprehensive trust, priority is given to identification, and for the data collected by data collection nodes with low comprehensive trust, the manual review criteria are more stringent.

[0050] Specifically, the comprehensive trust degree acquisition module includes:

[0051] Direct trust degree acquisition unit: used to acquire the direct trust degree of the current data collection node;

[0052] Recommendation trust degree acquisition unit: used to acquire the recommendation trust degree of other data collection nodes associated with the current data collection node for the current data collection node;

[0053] Comprehensive trust degree calculation unit: used to calculate the comprehensive trust degree of the current data collection node according to the direct trust degree and the recommendation trust degree.

Claims

1. A data collection and processing method, characterized in that, Including: Step 1: Obtain the comprehensive trustworthiness of data collection nodes; Step 2: Create a data recognition model to identify the authenticity of data; Step 3: Based on the data recognition model, identify the data collected by all data collection nodes and conduct manual review on the recognition results. For the data collected by data collection nodes with high comprehensive trustworthiness, identification is prioritized. For the data collected by data collection nodes with low comprehensive trustworthiness, the manual review criteria are more stringent.

2. The data collection and processing method according to claim 1, wherein The calculation method of the comprehensive trustworthiness is as follows: Step 11: Obtain the direct trustworthiness of the current data collection node and the recommended trustworthiness of other data collection nodes associated with the current data collection node for the current data collection node; Step 12: Calculate the comprehensive trustworthiness of the current data collection node according to the direct trustworthiness and the recommended trustworthiness.

3. The data collection and processing method according to claim 2, wherein The direct trustworthiness is determined according to the historical information of the current data collection node.

4. The data collection and processing method according to claim 2, wherein The calculation method of the recommended trustworthiness is: Determine the weights of each associated node according to the similarity with the current data collection node; Calculate the recommended trustworthiness according to the weights of each associated node and the direct trustworthiness of each associated node.

5. The data collection and processing method according to claim 2, wherein Comprehensive trust degree T total The calculation formula is: T total = w d *T d + w re *T re , where T d is the direct trust degree, T re is the recommended trust degree, w d and w re are the weights of the direct trust degree and the recommended trust degree respectively, and w d + w re = 1.

6. The data collection and processing method according to claim 1, characterized in that, The data recognition model includes recognition units created by multiple independent artificial intelligence algorithms, and the final output result of the data recognition model is the comprehensive value of the output results of each recognition unit.

7. The data collection and processing method according to any one of claims 1-6, characterized in that, It also includes re-obtaining the comprehensive trustworthiness of data collection nodes according to the trigger condition.

8. The data collection and processing method according to claim 7, wherein The trigger condition is time or event.

9. Data collection and processing system, characterized in that, Including: Comprehensive trustworthiness acquisition module: Used to obtain the comprehensive trustworthiness of data collection nodes; Model creation module: Used to create a data recognition model; Data processing module: Based on the data recognition model, identify the data collected by all data collection nodes and conduct manual review on the recognition results. For the data collected by data collection nodes with high comprehensive trustworthiness, identification is prioritized. For the data collected by data collection nodes with low comprehensive trustworthiness, the manual review criteria are more stringent.

10. The data collection and processing system according to claim 9, wherein The comprehensive trustworthiness acquisition module includes: Direct trustworthiness acquisition unit: Used to obtain the direct trustworthiness of the current data collection node; Recommended trustworthiness acquisition unit: Used to obtain the recommended trustworthiness of other data collection nodes associated with the current data collection node for the current data collection node; Comprehensive trustworthiness calculation unit: Used to calculate the comprehensive trustworthiness of the current data collection node according to the direct trustworthiness and the recommended trustworthiness.

Images