Data fusion system based on AI large model agent

Abstract

The invention relates to the technical field of data fusion, in particular to a data fusion system based on an AI large model agent, which comprises a feature extraction module, a feature analysis module, a tool module, an agent module, an AI large model module, a checking calculation module, an analysis module and an instruction control module. And the data fusion quality is judged based on the analysis module, and when the data fusion instruction does not reach the standard, the reason of data fusion failure is judged and targeted correction is carried out. According to the data fusion system provided by the invention, the original data can be efficiently acquired, the efficient and flexible data fusion process is ensured, meanwhile, the fused data quality is comprehensively evaluated, and a targeted optimization mechanism is provided, so that the problem that the quality does not reach the standard in the data fusion process is effectively solved, and the data fusion quality is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data fusion, and particularly relates to a data fusion system based on an AI large model agent. BACKGROUND

[0002] In the current era of rapid development of information technology, data sources are showing an explosive growth trend. In addition to traditional databases and information systems, emerging data sources such as Internet of Things devices, social media, and mobile devices are continuously generating massive amounts of data. These data from different sources each have their own characteristics and advantages, and can provide users with comprehensive and accurate decision-making basis. However, due to their significant differences in format, structure, and semantics, direct comprehensive analysis and application face many difficulties, which has prompted the emergence of data fusion technology.

[0003] The core of data fusion technology is to integrate data from different sources, formats, and structures. This technology can effectively eliminate the differences and barriers between data sources, and realize the interconnection and collaboration of data. However, it cannot be ignored that current data fusion methods are relatively weak in controlling the quality and reliability of fused data. This makes it difficult to fully realize the advantages of data fusion, thereby hindering the large-scale application and promotion of data fusion technology in more fields.

[0004] A data fusion method is proposed in Chinese patent CN119296328B, which obtains multiple traffic condition data sets collected on a target road section, and performs grid processing on two traffic condition data sets to be fused to obtain a space-time traffic condition graph corresponding to each traffic condition data set to be fused. Based on the data source corresponding to each traffic condition data set to be fused, the space-time traffic condition graph is fused to obtain a target space-time traffic condition graph of the target road section, and at least one first target congestion coordinate set is determined in the target space-time traffic condition graph through morphological operation and connected domain analysis. Or, at least one congestion coordinate set corresponding to each space-time traffic condition graph is fused to obtain at least one second target congestion coordinate set corresponding to the target road section space-time traffic condition graph. Although this method can realize the fusion of different data sets, it has obvious deficiencies in the evaluation and control of the quality of fused data. When the fusion result fails to meet the expected quality standard, this method lacks reliable evaluation indicators and is difficult to effectively modify the fusion process, thus cannot guarantee the reliability of the fused data, and further limits the credibility and effectiveness of the method in practical applications. SUMMARY

[0005] To this end, the application provides an AI large model agent-based data fusion system to solve the problems of lack of reliable evaluation standards after data fusion in the prior art and lack of methods for improving the reliability of fused data when the reliability of data fusion is insufficient.

[0006] In a first aspect, the application provides an AI large model agent-based data fusion system, comprising:

[0007] A feature extraction module is configured to receive a keyword;

[0008] A feature analysis module is connected to the feature extraction module and configured to obtain the name of a required tool based on the keyword;

[0009] A tool module contains a plurality of tools and is configured to access external resources related to the keyword;

[0010] An agent module is connected to the feature analysis module and the tool module and configured to call corresponding tools based on the determined tool name to obtain corresponding raw data;

[0011] An AI large model module is connected to the agent module and configured to receive the raw data and process and fuse the raw data, wherein the processing includes data duplicate value processing and data standardization;

[0012] A verification module is connected to the AI large model module and configured to verify the fused data;

[0013] An analysis module is connected to the verification module and configured to judge the data fusion quality based on the error degree of the fused data, determine the reason why the data fusion does not meet the standard based on the data error rate and a set data error rate difference when the data fusion quality is determined to be substandard, and re-determine the corresponding parameters in the fusion process based on the determined reason when the data fusion quality is determined to be substandard, including re-determining the judgment standard of the error degree or re-determining the data duplicate value processing standard, and issue a notification instruction when the data fusion quality is determined to be substandard after the parameter correction;

[0014] An instruction control module is connected to the analysis module and the AI large model module and configured to output the instruction output by the analysis module to the corresponding modules to make each module execute the corresponding instruction;

[0015] The tool is an API interface related to a content provider or a service provider.

[0016] Further, the analysis module is configured to determine whether the data fusion meets the standard based on the average error rate of the data, and, in the case that the data fusion is determined not to meet the standard, determine whether the data fusion meets the standard based on the missing degree of the fused data, or determine the reason why the data fusion does not meet the standard based on the average error rate of the data.

[0017] wherein the average error rate of the data is an average of error rates of each feature.

[0018] The feature is key information capable of representing the essential property of the original data in the original data.

[0019] The feature error rate is the error degree of each data source on the same feature, denoted as CR,

[0020] wherein n is the number of data sources in the original data, A i and B i represent two different measurement results of the same feature in the ith data source.

[0021] Further, the analysis module is configured to determine whether the data fusion meets the standard based on the missing rate, and, in the case that the data fusion is determined not to meet the standard, determine whether the data fusion meets the standard based on a ratio of the number of fields with conflicting field types and field value ranges in each of the fused data to the total number of fields in the fused data, or determine the reason why the data fusion does not meet the standard based on the average error rate of the data.

[0022] wherein the missing rate is an average of ratios of the missing byte amount of each of the fused data to the total byte amount before fusion.

[0023] Further, the analysis module is configured to determine whether the data fusion meets the standard based on the conflict ratio of each of the fused data, and, in the case that the data fusion is determined not to meet the standard, determine the reason why the data fusion does not meet the standard based on the average error rate of the data.

[0024] wherein the conflict ratio is a ratio of the number of fields with conflicting field types and field value ranges in the fused data to the total number of fields in the fused data.

[0025] Further, the analysis module is further configured to determine a reason for the data fusion not meeting the standard based on the average error rate of the data, and generate a corresponding processing instruction based on the determined reason, including: the analysis module determining the reason for the data fusion not meeting the standard based on a difference between the average error rate and a second preset average error rate, the data analysis module determining the reason for the data fusion not meeting the standard based on a variance of the data missing rate, or the analysis module determining that the current fusion algorithm cannot meet the demand based on the total amount of the original data correcting the preset average error rate.

[0026] Further, the analysis module is further configured to increase each of the preset average error rates based on the amount of the original data, and the increase amplitude of each of the preset average error rates is inversely proportional to the amount of the original data.

[0027] Further, the analysis module is further configured to determine whether the data fusion meets the standard based on each of the corrected preset average error rates, or determine that the current fusion algorithm cannot meet the demand based on the notification.

[0028] Further, the analysis module is further configured to determine a reason for the data fusion not meeting the standard based on the variance of the data missing rate, and determine the reason for the data fusion not meeting the standard based on a correlation degree between each of the original data and each of the fused data according to the reason for the data fusion not meeting the standard, or determine that the current fusion algorithm cannot meet the demand based on the notification.

[0029] Further, the analysis module is further configured to determine a reason for the data fusion not meeting the standard based on the correlation coefficient, and correct a data cleaning standard based on a data redundancy when it is determined that the fused data has redundancy, or determine that the current fusion algorithm cannot meet the demand based on the notification when it is determined that the fused data has no redundancy.

[0030] The correlation coefficient refers to a linear relationship strength between each of the original data and each of the fused data, denoted as PEX. The average value of the original data is denoted as PEX, The average value of the fused data is denoted as PEX.

[0031] Further, the analysis module is further configured to improve a data repetitive processing threshold based on the data redundancy, and the data repetitive processing threshold is inversely proportional to the data redundancy.

[0032] The data redundancy is a ratio between a repetitive or / and strongly correlated byte amount of each of the characteristic values and a total byte amount of the fused data.

[0033] ​Compared with the prior art, the data fusion system based on the AI large model agent can efficiently process complex multi-source data. Through the AI large model combined with the agent, the original data can be accurately obtained, and the flexibility and efficiency of the data fusion process are ensured. The data fusion quality is effectively guaranteed, and tasks such as data cleaning, standardization, and repeated value processing can be efficiently completed. In addition, the system is provided with an analysis module to judge the data fusion quality based on the error degree of the fused data. By setting clear error degree judgment standards, the fused data is comprehensively evaluated. This multi-dimensional evaluation standard makes the evaluation of data fusion quality more scientific and accurate, providing a reliable basis for subsequent data application. Finally, when the analysis module determines that the data fusion quality does not meet the standard, the corresponding parameters in the fusion process can be re-determined based on the determined reasons. If the data fusion quality still does not meet the standard after parameter correction, the system will issue a notification instruction to remind relevant personnel to take further measures. This correction mechanism effectively solves the problems that may occur in the data fusion process, ensures that the fused data meets the quality requirements, and significantly improves the overall quality of data fusion.

[0034] Further, the analysis module evaluates based on the average error rate and feature error rate of the data, which can accurately determine whether the data fusion meets the standard. When the fusion quality is found to be substandard, the reasons can be further analyzed in combination with the feature error rate to accurately locate the problem source. This accurate evaluation and positioning mechanism enables the system to correct parameters and optimize the fusion process, thereby improving the accuracy and consistency of the fused data and enhancing the reliability of the data.

[0035] Further, the analysis module comprehensively considers the missing rate and field conflict ratio to evaluate the data fusion quality from two dimensions of data integrity and consistency. When the fusion quality is found to be substandard, the reasons can be further analyzed and corresponding measures can be taken. This multi-dimensional evaluation method enables the system to timely discover and handle potential problems in data fusion, effectively improving the integrity and standardization of the fused data, and thus improving the reliability of the data.

[0036] Further, the analysis module comprehensively judges based on the conflict ratio and average error rate of the data to accurately identify the field conflict problem in data fusion. When the fusion quality is found to be substandard, the reasons can be analyzed in combination with the average error rate of the data and the parameters can be corrected accordingly. This comprehensive judgment mechanism effectively improves the consistency and accuracy of the fused data, further optimizes the data fusion process, and significantly enhances the reliability of the data.

[0037] Further, the analysis module comprehensively determines the reasons for the data fusion not meeting the standard through various ways, including error rate difference, data missing rate variance, etc., and accurately locates the problem source. Based on the determined reasons, targeted processing instructions are generated to timely adjust parameters or issue notifications to optimize the fusion process. This accurate positioning and targeted processing mechanism significantly improves the reliability and accuracy of the fused data, ensuring that the data fusion results meet the actual application requirements.

[0038] Further, the analysis module dynamically adjusts the preset average error rate according to the number of original data, and the increase amplitude is inversely proportional to the number of original data. This dynamic adjustment mechanism can flexibly adapt to the needs of data fusion of different scales, improving the accuracy of data fusion quality evaluation. By accurately adjusting the preset standard, the system can better handle fusion tasks under different data volumes, further improving the reliability and stability of the fused data, and meeting the diverse data application requirements.

[0039] Further, the analysis module determines whether the data fusion meets the standard based on the corrected preset average error rate, improving the accuracy and reasonableness of the evaluation. When the fusion quality is not up to standard, it can timely issue a notification to remind relevant personnel to take measures to further optimize the fusion process. This evaluation mechanism based on the corrected standard ensures the high-quality output of the fused data, significantly enhancing the reliability and credibility of the data, and providing a solid guarantee for the high-quality application of data.

[0040] Further, the analysis module combines the variance of the data missing rate and the correlation between the original data and the fused data to make a comprehensive judgment, accurately locating the reasons for the data fusion not meeting the standard. Through multi-dimensional analysis, the system can more comprehensively evaluate the data fusion quality and take targeted measures accordingly to optimize the fusion process. This comprehensive judgment mechanism effectively improves the accuracy and reliability of the fused data, ensuring that the data fusion results can better meet the actual application requirements.

[0041] Further, the analysis module makes a comprehensive judgment based on the correlation coefficient and data redundancy to accurately handle data redundancy problems. When it finds that the fused data has redundancy, it can correct the data cleaning standard based on the data redundancy to further improve data quality. This comprehensive judgment and correction mechanism effectively improves the purity and reliability of the fused data, ensuring that the data after fusion can have higher quality and credibility, providing high-quality data support for subsequent data analysis and decision-making.

[0042] Further, the analysis module dynamically adjusts the data repetition processing threshold according to the data redundancy, and the data repetition processing threshold is inversely proportional to the data redundancy. This dynamic adjustment mechanism can flexibly cope with different redundancy situations, effectively process data repetition values, and avoid the influence of redundant data on the quality of the fused data. By accurately adjusting the data repetition processing threshold, the system further improves the purity and reliability of the fused data, ensuring that the data after fusion can have higher quality and credibility, providing strong support for high-quality application of data. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 It is a module schematic diagram of the data fusion system based on the AI large model agent in the application;

[0044] Figure 2 It is a flow schematic diagram of the data fusion system based on the AI large model agent in the application;

[0045] Figure 3 It is a logic determination diagram for determining whether the data fusion meets the standard based on the average data error rate in the application;

[0046] Figure 4 It is a logic determination diagram for determining the reason why the data fusion does not meet the standard based on the data error rate difference in the application. DETAILED DESCRIPTION

[0047] In order to make the objects and advantages of the present application clearer, the present application will be further described below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0048] The preferred embodiments of the present application will be described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and do not limit the protection scope of the present application.

[0049] Please refer to Figure 1 shown, which is a module schematic diagram of the data fusion system based on the AI large model agent in the embodiment of the application. The system described in the embodiment of the application comprises a feature extraction module, a feature analysis module, a tool module, an agent module, an AI large model module, a checking module, an analysis module and an instruction control module. Among them:

[0050] The feature extraction module is used to receive a keyword;

[0051] The feature analysis module is connected with the feature extraction module and is used to obtain the name of the required tool based on the keyword;

[0052] The tool module comprises a plurality of tools and is used to access external resources related to the keyword;

[0053] An agent module connected with the feature analysis module and the tool module, used to call the corresponding tool based on the determined tool name to obtain the corresponding raw data;

[0054] An AI large model module connected with the agent module, used to receive the raw data, and used to process and fuse the raw data, wherein the processing includes data duplicate value processing and data standardization;

[0055] A verification module connected with the AI large model module, used to verify the fused data;

[0056] An analysis module connected with the verification module, used to judge the data fusion quality based on the error degree of the fused data, and used to determine the reason for the data fusion not meeting the standard based on the data error rate and the set data error rate difference in the case of determining that the data fusion quality does not meet the standard, and used to re-determine the corresponding parameters in the fusion process, including re-determining the judgment standard of the error degree or re-determining the data duplicate value processing standard, in the case of determining that the data fusion quality does not meet the standard after parameter correction, and used to issue a notification instruction;

[0057] An instruction control module connected with the analysis module and the AI large model module, used to output the instruction output by the analysis module to the corresponding module to make each module execute the corresponding instruction.

[0058] The tool is an API interface related to a content provider or a service provider.

[0059] Specifically, in the process of processing the raw data by the AI large model module, first, each raw data is standardized into a vector by fuzzy C-means clustering standardization, and the specific steps include:

[0060] Set the number of clusters C as one-half of the square root of the raw data, set the fuzzy coefficient m as 2, set the iteration data processing threshold S, the specific value of the iteration data processing threshold S can be summarized according to the actual situation and the law of each historical data, initialize the membership matrix U, and the element U ij represents the membership of the ith data point to the jth cluster. Ensure that the sum of the membership of each row is 1, i = 1, 2, 3, ···, n, j = 1, 2, 3, ···, n;

[0061] Calculate the center of each cluster according to the membership matrix, and the calculation formula is:

[0062]

[0063] where N is the number of data points, X i is the i-th data point, C j is the center of the j-th cluster.

[0064] The membership matrix is updated according to the cluster centers. The update formula is:

[0065]

[0066] where ||X i -C k || is the Euclidean distance from the i-th data point to the center of the j-th cluster.

[0067] The difference between the new membership matrix and the old membership matrix is calculated. If the difference is less than S, the iteration is stopped; otherwise, the iteration is continued according to the membership matrix to calculate the center of each cluster.

[0068] After standardization, extract 10% of the data in the original data at random and detect the repetition rate, set the data repetition processing threshold M equal to the repetition rate obtained by detection, and then judge the similarity between each of the original data in turn by cosine similarity. The specific steps of judging the similarity between the original data by cosine similarity include: obtaining the standardized original data in turn, and calculating the cosine value of the included angle by the cosine similarity formula, the calculation formula is: where X and Y represent two pieces of standardized original data, |X| and |Y| represent the modulus of the corresponding data, theta represents the included angle of the two data in space, and cos theta is the similarity between the data X and Y.

[0069] If the similarity between the two data is greater than M, it is considered that the data is repeated, and the detected data is deleted, and the original data is traversed.

[0070] Referring to Figure 2 The AI large model agent-based data fusion system of the embodiment of the application is shown in the figure, which is a process schematic diagram of the AI large model agent-based data fusion system of the embodiment of the application. When the AI large model agent-based data fusion system of the embodiment of the application is running, the feature extraction module accepts the keyword sent by the user and sends it to the feature analysis module. The feature analysis module determines the tool name required to complete the keyword according to the keyword. The tool refers to the API interface related to the content provider or the service provider. The agent module receives the tool name and calls the tool module to obtain the original data. The AI large model module accepts the original data output by the agent, then standardizes the original data by the fuzzy C-means clustering standardization algorithm, and then removes the repeated data in the standardized data by cosine similarity.

[0071] The AI large model module fuses the processed data, and calculates parameters of the fused data through the verification module to judge the data fusion quality, the parameters including: data average error rate, missing rate, conflict ratio, error rate difference, data missing rate variance, and correlation coefficient.

[0072] The analysis module judges the data fusion quality according to the data average error rate, and judges the reason for the data fusion not meeting the standard based on the data error rate difference in the case of data not meeting the standard, and then determines corresponding parameters in the fusion process or determines to issue a notification that the current fusion algorithm cannot meet the requirements according to the reason for the data not meeting the standard. The instruction control module outputs the instruction to the corresponding module after receiving the instruction of the analysis module, so that the module executes the corresponding instruction.

[0073] Further, the analysis module is used to judge whether the data fusion meets the standard based on the data average error rate, and to judge whether the data fusion meets the standard based on the missing degree of the fused data or to judge the reason for the data fusion not meeting the standard based on the data average error rate in the case of judging that the data fusion does not meet the standard.

[0074] The data average error rate is an average value of each feature error rate.

[0075] The feature is key information capable of representing the essential property of the original data in the original data.

[0076] The feature error rate is the error degree of each data source on the same feature, denoted as CR,

[0077] Wherein, n is the data of the data source in the original data, the unit is piece, A i and B i represent two different measurement results of the same feature in the i-th data source.

[0078] Specifically, please refer to Figure 3 , which is a logic judgment diagram for judging whether the data fusion meets the standard based on the data average error rate in the embodiment of the application. The data average error rate calculates an average value of each feature error rate, which can intuitively represent the overall error degree of the data before and after fusion. If the data average error rate is low, it indicates that the data after fusion is closer to the original data in each feature, and the accuracy of the data is higher. On the contrary, if the data average error rate is high, it indicates that the overall data after fusion has a large deviation from the original data, and the accuracy of the data is lower. By taking the average value, the one-sided influence of a single feature error on the overall quality evaluation result can be avoided, and the evaluation result can more comprehensively and objectively reflect the overall quality level of the data after fusion.

[0079] The process that the analysis module uses to determine whether the data fusion meets the standard based on the average error rate A includes:

[0080] The analysis module obtains the average error rate A of the data and sets a first preset average error rate A1 and a second preset average error rate A2, where A1∈[0.5, 8%] and A2∈(8, 16%].

[0081] If the average error rate A of the data is less than or equal to the first preset average error rate A1, the analysis module determines that the quality of the current data after fusion meets the standard and outputs the result required by the user based on the fused data.

[0082] If the average error rate A of the data is greater than the first preset average error rate A1 and less than or equal to the second preset average error rate A2, the analysis module determines whether the current data fusion meets the standard in combination with the degree of loss of the fused data.

[0083] If the average error rate A of the data is greater than the second preset average error rate A2, the analysis module determines that the current fused data cannot meet the requirement, and determines the reason for the failure of the current data fusion in combination with the difference between the average error rate A of the data and the second preset average error rate A2.

[0084] Further, the analysis module uses the loss rate to determine whether the data fusion meets the standard, and in the case where it is determined that the data fusion does not meet the standard, uses the ratio of the number of fields in which the field type and the field value range conflict to the total number of fields in the fused data to determine whether the data fusion meets the standard, or determines the reason for the failure of the data fusion based on the average error rate.

[0085] The loss rate is the average of the ratio of the amount of lost bytes to the total amount of bytes before fusion.

[0086] Specifically, the analysis module uses the loss rate to determine whether the data fusion meets the standard. The loss rate is the average of the ratio of the amount of lost bytes to the total amount of bytes before fusion. Through the loss rate, the degree of loss before and after the current data fusion can be effectively measured, so as to further determine the quality of the current fused data. The steps that the analysis module uses the loss rate to determine whether the data fusion meets the standard include:

[0087] The analysis module obtains the loss rate D and sets a first loss rate D1, where D1∈

[0088] [0.1, 10%].

[0089] The loss rate D is compared with the first loss rate D1.

[0090] If the missing rate D is less than or equal to the first missing rate D1, the analysis module determines whether the data fusion meets the standard by combining the ratio of the number of fields in each of the fused data in which the field type and the field value range conflict to the total number of fields in the fused data;

[0091] If the missing rate D is greater than the first missing rate D1, the analysis module determines that the current fused data cannot meet the requirement, and the analysis module determines the reason for the failure of the current data fusion by combining the difference between the average error rate A and the second preset average error rate A2.

[0092] Further, the analysis module is used to determine whether the data fusion meets the standard based on the conflict ratio of each of the fused data, and determine the reason for the data fusion not meeting the standard based on the average error rate of the data in the case of determining that the data fusion does not meet the standard;

[0093] The conflict ratio is the ratio of the number of fields in the fused data in which the field type and the field value range do not match to the total number of fields in the fused data. The conflict ratio is calculated by calculating the proportion of the number of fields in the fused data in which the field type and the value range do not match to the total number of fields, to quantify the consistency problem of the data. When the conflict ratio is small, it indicates that the types and value ranges of most fields are matched, and the data maintains a high consistency in semantics and structure. Conversely, a larger conflict ratio means that there are more fields with type or value range conflicts, and the data consistency is poor. This index can effectively capture potential problems caused by inconsistent field definitions in the data fusion process, and provides a key basis for optimizing the data fusion process and improving data quality.

[0094] Specifically, the step of the analysis module determining whether the data fusion meets the standard based on the conflict ratio of each of the fused data includes:

[0095] The analysis module obtains the conflict ratio E of the fused data and sets the first conflict ratio E1. The first conflict ratio E1 is set as the iteration data processing threshold S multiplied by 10 4 ;

[0096] The conflict ratio E is compared with the first conflict ratio E1:

[0097] If the conflict ratio E is less than or equal to the first conflict ratio E1, the analysis module determines that the quality of the current data fusion meets the standard, and outputs the result required by the user according to the fused data;

[0098] If the conflict ratio E is greater than the first conflict ratio E1, the analysis module determines that the current fused data cannot meet the requirements, and the analysis module determines the reason for the failure of the current data fusion in combination with the difference between the data average error rate A and the second preset average error rate A2.

[0099] Further, the analysis module is also used to determine the reason for the data fusion not meeting the standard based on the data average error rate, and generate corresponding processing instructions based on the determined reason, including: the analysis module determines the reason for the data fusion not meeting the standard based on the error rate difference, the data analysis module determines the reason for the data fusion not meeting the standard based on the variance of the data missing rate, or the analysis module determines to correct the preset average error rate based on the total amount of the original data, or the analysis module determines to issue a notification that the current fusion algorithm cannot meet the requirements;

[0100] The error rate difference is the difference between the data average error rate and the second preset average error rate, and the error rate difference reflects the deviation degree between the actual data average error rate and the preset standard. According to the size, the application can accurately locate the specific reason for the data fusion not meeting the standard, and implement corresponding strategies, thereby significantly enhancing the reliability of the fused data.

[0101] Specifically, please refer to Figure 4 The analysis module is used to determine the reason for the data fusion not meeting the standard based on the data error rate difference B, and the process includes:

[0102] The analysis module obtains the data error rate difference B, and compares the data error rate difference B with the first preset error rate difference B1 and the second preset error rate difference B2 set, the first preset error rate difference B1 ∈ [0.1, 10%], and the second preset error rate difference B2 is set as (10%, 20%];

[0103] If the data error rate difference B is less than or equal to the first preset error rate difference B1, the analysis module increases each of the preset average error rates based on the number of the original data;

[0104] If the data error rate difference B is greater than the first preset error rate difference B1 and less than or equal to the second preset error rate difference B2, the reason for the data fusion not meeting the standard is determined in combination with the variance of the data missing rate;

[0105] If the data error rate difference B is greater than the second preset error rate difference B2, the analysis module determines to issue a notification that the current fusion algorithm cannot meet the requirements.

[0106] Further, the analysis module is configured to increase each of the preset average error rates based on the amount of original data, and the increase range of each of the preset average error rates is inversely proportional to the amount of original data.

[0107] With the same amount of error data bytes, a smaller amount of original data may lead to a larger error rate. In order to ensure the reliability of the evaluation standard, it is necessary to adjust each of the preset average error rates according to the amount of original data, so that each of the preset average error rates can adapt to different data amount scenarios, thereby improving the accuracy of the evaluation.

[0108] Specifically, the process of increasing each of the preset average error rates based on the amount of original data includes:

[0109] The analysis module obtains the amount of original data G, and compares the amount of original data G with a preset first data amount G1 and a preset second data amount G2, wherein G1 ∈ (0, 800MB] and G2 ∈ (800, 1024MB].

[0110] [50,800MB] and G2 ∈ (800, 1024MB];

[0111] If the amount of original data G is less than or equal to the preset first data amount G1, the second preset average error rate A2 is corrected to a corresponding value using a first average error rate correction value a1, and the corrected second preset average error rate A2' = A2 x a1, wherein the first average error rate correction value a1 is set to 1.23.

[0112] If the amount of original data G is greater than the preset first data amount G1 and less than or equal to the preset second data amount G2, the second preset average error rate A2 is corrected to a corresponding value using a second average error rate correction value a2, and the corrected second preset average error rate A2' = A2 x a2, wherein the second average error rate correction value a2 is set to 1.18.

[0113] If the amount of original data G is greater than the preset second data amount G2, the second preset average error rate A2 is corrected to a corresponding value using a third average error rate correction value a3, and the corrected second preset average error rate A2' = A2 x a3, wherein the third average error rate correction value a3 is set to 1.1.

[0114] Further, the analysis module is configured to determine whether the data fusion meets the standard based on the corrected each of the preset average error rates, or the analysis module determines to issue a notification that the current fusion algorithm cannot meet the demand.

[0115] Specifically, in the embodiment, the process of determining whether the data fusion meets the standard based on the corrected each of the preset average error rates includes:

[0116] The corrected second preset average error rate A2' is obtained, and the corrected second preset average error rate A2' is compared with the data average error rate A;

[0117] If A is less than or equal to A2', the analysis module considers that the current data fusion meets the standard;

[0118] If A is greater than A2', the analysis module determines to issue a notification that the current fusion algorithm cannot meet the requirements.

[0119] Further, the analysis module is used to determine the reason why the data fusion does not meet the standard based on the variance of the data missing rate, and determine the reason why the data fusion does not meet the standard based on the correlation degree of each original data and each fused data, or the analysis module determines to issue a notification that the current fusion algorithm cannot meet the requirements.

[0120] The variance of the data missing rate reflects the fluctuation of the data missing degree. A small variance indicates that the data missing situation is stable, which usually means that the data structure is consistent and the data amount is sufficient. If the variance is large, there is obvious fluctuation, which may indicate that the data is insufficient or the structure is unstable. However, when the data amount is sufficient and the structure is stable, but still cannot meet the requirements after data fusion, it means that there may be a high similarity between the data, and at this time, the correlation coefficient needs to be combined to determine whether data redundancy has occurred.

[0121] Further, the analysis module is used to determine the reason why the data fusion does not meet the standard based on the correlation coefficient, and correct the data duplicate value processing standard based on the data redundancy in the case where the fused data is determined to have redundancy, and issue a notification that the current fusion algorithm cannot meet the requirements in the case where the fused data is determined to have no redundancy;

[0122] The correlation coefficient refers to the linear relationship strength between each original data and each fused data, denoted as PEX, is the average value of the original data, is the average value of the fused data, and the correlation coefficient is used to quantify the linear relationship strength and direction between two data sets. In the data fusion scenario, the correlation coefficient of the original data and the fused data can directly reflect the similarity between the two, and redundant data is often repeated or highly correlated data. When redundancy occurs after data fusion, the information increment in the fused data is relatively small, and the overlap with the original data is high. In this case, the correlation coefficient of the fused data and the original data will significantly increase. Therefore, by calculating the correlation coefficient of the original data and the fused data, it can be effectively determined whether redundancy occurs after data fusion.

[0123] ​Further, the analysis module is configured to increase the data repetition processing threshold based on the data redundancy, and the data repetition processing threshold is inversely proportional to the data redundancy.

[0124] The data redundancy is a ratio between the amount of repeated and / or strongly correlated bytes of each feature value and the total amount of bytes of the fused data.

[0125] Specifically, in the embodiment, the analysis module determines the reason for the data fusion not meeting the standard based on the variance K of the data missing rate, and the process includes:

[0126] The analysis module compares the variance K of the data missing rate with a preset missing rate variance K1, and the specific value of the missing rate variance K1 can be obtained according to the actual situation and the law of each historical data:

[0127] If the variance K of the data missing rate is less than or equal to the preset missing rate variance K1, the analysis module determines that the fused data has redundancy, and the analysis module determines the reason for the data fusion not meeting the standard based on the correlation degree of each original data and each fused data.

[0128] If the variance K of the data missing rate is greater than the preset missing rate variance K1, the analysis module determines to send a notification that the current fusion algorithm cannot meet the demand.

[0129] The analysis module determines the reason for the data fusion not meeting the standard based on the correlation degree of each original data and each fused data, and the process includes:

[0130] The calculation module calculates the correlation coefficient PEX of each original data and each fused data, and the analysis module compares the correlation coefficient PEX of each original data and each fused data with a preset correlation coefficient PEX1 before and after fusion, and the correlation coefficient PEX1 is in the range of [0.5, 1].

[0131] If the correlation coefficient PEX is greater than PEX1, it indicates that the data before and after fusion has a high correlation degree, but the current fusion quality cannot meet the requirements, which indicates that there are many data with strong correlation but not completely repeated at this time, that is, there is a correlation redundancy phenomenon, which leads to the difficulty of meeting the fusion standard of the fused data, and therefore it is necessary to further improve the data repetition value processing standard to avoid the generation of the correlation redundancy phenomenon. At this time, the analysis module determines that the current fused data has redundancy, and the analysis module corrects the data repetition value processing standard based on the data redundancy.

[0132] If the correlation coefficient PEX is less than or equal to PEX1, the analysis module determines that the current fused data does not have data redundancy, and the analysis module determines to send a notification that the current fusion algorithm cannot meet the demand.

[0133] The process of improving the data repetition value processing standard based on the data redundancy by the analysis module comprises:

[0134] The fusion data redundancy L is obtained by the calculation module, and the analysis module compares the data redundancy L with the set first preset data redundancy L1 and second preset data redundancy L2, wherein the first preset data redundancy L1 is set to be in the range of [0.1, 0.4], and the second preset data redundancy L2 is set to be in the range of (0.4, 0.8];

[0135] If the data redundancy L is less than or equal to the first preset data redundancy L1, the data repetition processing threshold M is corrected to a corresponding value by using a first redundancy correction value β1, and the corrected data repetition processing threshold M' = M x β1, wherein the first redundancy correction value β1 is set to be 1.06;

[0136] If the data redundancy L is greater than the first preset data redundancy L1 and less than or equal to the second preset data redundancy L2, the data repetition processing threshold M is corrected to a corresponding value by using a second redundancy correction value β2, and the corrected data repetition processing threshold M' = M x β2, wherein the second redundancy correction value β2 is set to be 1.1;

[0137] If the data redundancy L is greater than the second preset data redundancy L2, the data repetition processing threshold M is corrected to a corresponding value by using a third redundancy correction value β3, and the corrected data repetition processing threshold M' = M x β3, wherein the third redundancy correction value β3 is set to be 1.15.

[0138] The technical scheme of the present application has been described in combination with the preferred embodiments shown in the drawings, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the present application, and the technical schemes after the changes or replacements will all fall within the protection scope of the present application.

Claims

1. An AI large model-based agent data fusion system, characterized in that, The application comprises: a feature extraction module for receiving keywords; a feature analysis module connected to the feature extraction module for obtaining the name of a required tool based on the keywords; a tool module containing a plurality of tools for accessing external resources related to the keywords; an agent module connected to the feature analysis module and the tool module for calling corresponding tools based on the determined tool name to obtain corresponding raw data; an AI large model module connected to the agent module for receiving the raw data and processing and fusing the raw data, wherein the processing includes data duplicate value processing and data standardization; a verification module connected to the AI large model module for verifying the fused data; an analysis module connected to the verification module for judging the data fusion quality based on the error degree of the fused data, determining the reason for the data fusion not meeting the standard based on the data error rate and the set data error rate difference in the case of determining that the data fusion quality does not meet the standard, and re-determining the corresponding parameters in the fusion process based on the determined reason in the case of determining that the data fusion quality does not meet the standard, including re-determining the error degree judgment standard or re-determining the data duplicate value processing standard, and issuing a notification instruction in the case of determining that the data fusion quality does not meet the standard after parameter correction; an instruction control module connected to the analysis module and the AI large model module for outputting the instruction output by the analysis module to the corresponding module to make each module execute the corresponding instruction; the tool is an API interface related to a content provider or a service provider. 2.The AI large model-based agent's data fusion system according to claim 1, wherein, The analysis module is used to determine whether the data fusion meets the standard based on the average data error rate, and to determine whether the data fusion meets the standard based on the missing degree of the fused data or the reason for the data fusion not meeting the standard based on the average data error rate in the case of determining that the data fusion does not meet the standard; wherein the average data error rate is the average of each feature error rate, the feature is the key information representing the essential attribute of the raw data in the raw data, and the feature error rate is the error degree of each data source on the same feature. 3.The AI large model-based agent's data fusion system according to claim 2, characterized in that, The analysis module is used to determine whether the data fusion meets the standard based on the missing rate, and to determine whether the data fusion meets the standard based on the ratio of the number of fields with conflicting field types and field value ranges in each of the fused data to the total number of fields in the fused data or the reason for the data fusion not meeting the standard based on the average data error rate in the case of determining that the data fusion does not meet the standard; wherein the missing rate is the average of the ratio of the missing byte amount of each of the fused data to the total byte amount before fusion. 4.The AI large model-based agent's data fusion system according to claim 3, wherein, The analysis module is used to determine whether the data fusion meets the standard based on the conflict ratio of each of the fused data, and to determine the reason for the data fusion not meeting the standard based on the average data error rate in the case of determining that the data fusion does not meet the standard. The conflict ratio is a ratio of a number of fields in the fused data in which a field type does not match a field value range to a total number of fields in the fused data. 5.The AI large model-based agent's data fusion system according to claim 2, wherein, The analysis module is further configured to determine a reason for the data fusion not meeting the standard based on the average error rate of the data, and generate a corresponding processing instruction based on the determined reason, including: the analysis module determining the reason for the data fusion not meeting the standard based on a difference between the average error rate and a second preset average error rate, the data analysis module determining the reason for the data fusion not meeting the standard based on a variance of the data missing rate, or the analysis module determining that the current fusion algorithm cannot meet the demand based on a correction of the preset average error rate by the total amount of the original data, wherein the difference between the average error rate and the second preset average error rate is a difference between the average error rate of the data and the second preset average error rate. 6.The AI large model-based agent's data fusion system according to claim 5, wherein, The analysis module is configured to increase each of the preset average error rates based on the total amount of the original data, and the increase amplitude of each of the preset average error rates is inversely proportional to the total amount of the original data. 7.The AI-large model based agent's data fusion system according to claim 6, characterized in that, The analysis module is configured to determine whether the data fusion meets the standard based on the corrected each of the preset average error rates, or determine that the current fusion algorithm cannot meet the demand based on a notification. 8.The AI large model based agent's data fusion system according to claim 5, wherein, The analysis module is configured to determine a reason for the data fusion not meeting the standard based on the variance of the data missing rate, and determine the reason for the data fusion not meeting the standard based on a correlation degree between each of the original data and each of the fused data according to the reason for the data fusion not meeting the standard, or determine that the current fusion algorithm cannot meet the demand based on a notification. 9.The AI-large model based agent's data fusion system according to claim 8, wherein, The analysis module is configured to determine the reason for the data fusion not meeting the standard based on the correlation coefficient, and correct a data cleaning standard based on a data redundancy when it is determined that the fused data has redundancy, or determine that the current fusion algorithm cannot meet the demand based on a notification when it is determined that the fused data has no redundancy. The correlation coefficient refers to a linear relationship strength between each of the original data and each of the fused data. 10.The AI large model based agent's data fusion system according to claim 9, characterized in that, The analysis module is configured to improve a data repetition processing threshold based on the data redundancy, and the data repetition processing threshold is inversely proportional to the data redundancy. The data redundancy is a ratio between a repeated and / or strongly correlated byte amount of each of the characteristic values and a total byte amount of the fused data.

Images