A computer data processing system based on machine learning

Abstract

The present invention relates to the field of data processing technology, and in particular to a computer data processing system based on machine learning, comprising: a multi-source data acquisition module for real-time acquisition of multi-source heterogeneous data and source identification; a data timestamp construction module for numbering the multi-source heterogeneous data according to the acquisition time to obtain a timestamp number; a data multi-dimensional classification module for determining the comprehensive type of data; a type feature analysis module for performing feature analysis to obtain data features; a priority scoring module for updating a three-dimensional classification model; a multi-source data processing module for responding to and processing the multi-source heterogeneous data in accordance with the comprehensive type of data and the priority score sequence; and a processing anomaly monitoring module for counting the number of anomaly processing times of the multi-source heterogeneous data and performing comparative analysis on the anomaly processing times to trigger rule optimization of the three-dimensional classification model. The present invention implements priority processing of multi-source heterogeneous data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a computer data processing system based on machine learning. Background Art

[0002] With the development of information technology, a large amount of multi-source heterogeneous data has been generated in fields such as manufacturing. These data have different structures and sources, so it is necessary to scientifically allocate resources and optimize processing processes to determine reasonable processing priorities.

[0003] Chinese Patent Publication No.: CN104798043A discloses a data processing method and computer system, including discretizing data samples to obtain data samples in matrix form (S101), training the data samples in matrix form according to a preset classification method to obtain a classification rule set (S102), and converting the classification rule set into a classification rule set recognized by a data decision platform (S103), and providing it to the data decision platform (S104), so that the data decision platform can make data decisions based on the classification rule set recognized by the data decision platform obtained by the computer system. The above process from training the data samples to applying the trained classification rule set to data decision making is automatically completed by the computer system, avoiding manual participation. When the data samples change or the original classification rule set needs to be updated, the updated classification rule set can be obtained in time. This invention realizes the classification processing training of sample data, but does not realize the comprehensive priority processing and analysis of multi-source heterogeneous data. There are problems such as low efficiency in analyzing multi-source heterogeneous data and inaccurate priority processing of multi-source heterogeneous data. Summary of the Invention

[0004] The object of the present invention is to provide a computer data processing system based on machine learning to solve at least one of the problems existing in the prior art.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A computer data processing system based on machine learning, comprising:

[0007] The data multi-dimensional classification module is used to classify the security level of the data content of multi-source heterogeneous data and build a three-dimensional classification model based on the real-time nature, data volume and security level of multi-source heterogeneous data to determine the comprehensive type of data;

[0008] Type feature analysis module, used to perform feature analysis on data comprehensive type, timestamp number and multi-source heterogeneous data to obtain data features;

[0009] The priority scoring module is used to weight the real-time nature, data volume, security level and data characteristics of multi-source heterogeneous data to obtain a priority score. It is also used to match and analyze the priority score with the comprehensive data type to update the three-dimensional classification model.

[0010] Furthermore, the system further comprises:

[0011] Multi-source data acquisition module, used to collect multi-source heterogeneous data and source identification in real time;

[0012] The data timestamp construction module is used to number multi-source heterogeneous data according to the collection time to obtain the timestamp number;

[0013] Multi-source data processing module, used to respond and process multi-source heterogeneous data according to the data comprehensive type and priority score order;

[0014] The anomaly processing monitoring module is used to count the number of anomaly processing times of multi-source heterogeneous data and perform comparative analysis on the anomaly processing times to trigger rule optimization of the three-dimensional classification model.

[0015] Furthermore, the data multi-dimensional classification module is provided with a security level division unit, which is used to divide the security level of the multi-source heterogeneous data according to the data content. The security level division unit divides the security level of the multi-source heterogeneous data into three levels according to the different data contents;

[0016] The data multi-dimensional classification module is further provided with a data volume grading unit, which is used to compare and analyze the data volume of multi-source heterogeneous data to divide the data volume level. The data volume grading unit divides the multi-source heterogeneous data into three levels according to the size of the data volume;

[0017] The data multi-dimensional classification module is further provided with a time sensitivity analysis unit, which is used to classify delay tolerance levels according to source identification. The time sensitivity analysis unit classifies the delay tolerance levels of multi-source heterogeneous data into three levels according to different source identifications.

[0018] Furthermore, the data multi-dimensional classification module is further provided with a three-dimensional classification construction unit, which is used to encode the delay tolerance level, data volume level and security level to obtain a digital level, and quantitatively represent the delay tolerance level, data volume level and security level;

[0019] The three-dimensional classification construction unit analyzes the type judgment parameters according to the digital level of the delay tolerance level, data volume level and security level of the multi-source heterogeneous data to obtain the type judgment parameter S, and compares and analyzes the type judgment parameters to judge the data synthesis type, and divides the data synthesis type of the multi-source heterogeneous data into four categories according to the size of the type judgment parameters. The data synthesis types include emergency security affairs, emergency standard affairs, conventional standard affairs and batch processing affairs.

[0020] Furthermore, the type feature analysis module performs feature analysis on the data comprehensive type, timestamp number and multi-source heterogeneous data to obtain the data feature L(d).

[0021] Furthermore, the priority scoring module is provided with a data scoring analysis unit, which is used to analyze the grade scores according to the delay tolerance level, data volume level and security level of the multi-source heterogeneous data to obtain a security level score B1, a data volume level score B2 and a delay tolerance level B3;

[0022] The priority scoring module is also provided with a scoring weighted analysis unit, which is used to perform weighted analysis on the grade score and data features to obtain a priority score. The priority score is set to Z. If L(d)>θ, set Z=β1×B1+β2×B2+β3×B3; otherwise, set Z=[β1+L(d) / 10]×B1+[β2-L(d) / 5]×B2+[β3+L(d) / 10]×B3, wherein β1 represents the first weight, β2 represents the second weight, β3 represents the third weight, β1+β2+β3=1 and β1×β2×β3≠0.

[0023] Furthermore, the priority scoring module is further provided with a classification matching update unit, which is used to perform matching analysis on the priority score and the data comprehensive type, extract the priority score corresponding to different data comprehensive types as the comprehensive type score, and arrange the comprehensive type scores corresponding to each data comprehensive type in order from small to large, and analyze the type discrete parameter according to the arranged comprehensive type score to obtain the type discrete parameter R(m);

[0024] The classification matching update unit compares and analyzes the discrete parameters of the type to update the three-dimensional classification model. When R(m)>r, the classification matching update unit matches the discrete parameters of the type to update the three-dimensional classification model. 0.5×NU(m) +Min(Z∈U(m)) adds a priority tag to the multi-source heterogeneous data, and r represents the updated matching threshold.

[0025] Furthermore, the multi-source data processing module responds to and processes the multi-source heterogeneous data according to the data integration type and priority score order. The multi-source heterogeneous data processing module sorts the multi-source heterogeneous data with the same data integration type and with priority tags and without priority tags in descending order according to the priority score, and preferentially processes the multi-source heterogeneous data with priority tags in the order of arrangement, and then processes the multi-source heterogeneous data without priority tags in the order of arrangement. If Z>z1, the multi-source data processing module allocates the multi-source heterogeneous data to the GPU acceleration queue for processing. If Z<z2, the multi-source data processing module classifies the multi-source heterogeneous data into the night batch task pool for processing. If z2≤Z≤z1, the multi-source data processing module processes the multi-source heterogeneous data normally; wherein z1 represents the first processing threshold and z2 represents the second processing threshold.

[0026] Furthermore, the processing abnormality monitoring module is provided with a timeout processing judgment unit, which is used to judge the completion type of the multi-source heterogeneous data according to the processing time, timestamp number and delay tolerance level. When the delay tolerance level is level one, if T≤γ1, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is normal completion; if T>γ1, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is timeout completion; if the processing time is a missing value, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is incomplete; when the delay tolerance level is level one, if T≤γ1, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is normal completion; if T>γ1, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is timeout completion; if the processing time is a missing value, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is incomplete; When the delay tolerance level is level two, if T≤γ2, the timeout processing judgment unit determines that the completion type of the multi-source heterogeneous data is normal completion; if T>γ2, the timeout processing judgment unit determines that the completion type of the multi-source heterogeneous data is timeout completion; if the processing time is a missing value, the timeout processing judgment unit determines that the completion type of the multi-source heterogeneous data is incomplete; when the delay tolerance level is level three, the timeout processing judgment unit does not judge the completion type; wherein T represents the processing time, γ1 represents the first delay threshold, and γ2 represents the second delay threshold.

[0027] Furthermore, the processing exception monitoring module is further provided with an exception processing analysis unit, which is used to count the number of completion types of timeout completion as the number of timeout processing times, count the number of completion types of incomplete completion as the number of incomplete completion times, and take the sum of 70% of the timeout processing times and the incomplete processing times as the number of exception processing times;

[0028] When the number of timeout processing times of multi-source heterogeneous data of the same data integration type in one second is greater than or equal to k times, the exception handling analysis unit triggers the rule optimization of the three-dimensional classification model. The rule optimization of the three-dimensional classification model is to optimize the weighted analysis process of the grade score according to the number of timeout processing times of multi-source heterogeneous data of the same data integration type in one second, optimize the weight of each grade score, and logarithmically correlate the weight of each grade score with the number of timeout processing times to optimize the three-dimensional classification model.

[0029] The beneficial effects of this invention are as follows: Through the collaborative work of various units, efficient processing and dynamic optimization of multi-source heterogeneous data are achieved. Security level division, data volume classification, and time sensitivity analysis ensure the rational classification and priority allocation of data; three-dimensional classification construction and weighted scoring analysis improve the accuracy and efficiency of data processing; anomaly monitoring and rule optimization mechanisms enhance the system's adaptability and stability, improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 Schematic diagram of the structure of the computer data processing system based on machine learning in this embodiment.

[0032] Figure 2 This is a structural diagram of the data multi-dimensional classification module in this embodiment.

[0033] Figure 3 Schematic diagram of the structure of the priority scoring module in this embodiment.

[0034] Figure 4 This is a schematic diagram of the structure of the abnormality monitoring module in this embodiment. DETAILED DESCRIPTION

[0035] In order to more clearly illustrate the present invention, the present invention is further described below in conjunction with preferred embodiments and accompanying drawings. Similar components in the accompanying drawings are represented by the same reference numerals. It should be understood by those skilled in the art that the following detailed description is illustrative rather than restrictive and should not be used to limit the scope of protection of the present invention.

[0036] It should be noted that, although the terms "first," "second," and "third" may be used to describe the embodiments of the present application, the description should not be limited to these terms. These terms are merely used to distinguish the descriptions. For example, without departing from the scope of the embodiments of the present application, "first" may also be referred to as "second," and similarly, "second" may also be referred to as "first."

[0037] See also Figure 1 As shown, this is a computer data processing system based on machine learning in this embodiment, including:

[0038] The multi-source data acquisition module is used to collect multi-source heterogeneous data and source identification in real time. The multi-source heterogeneous data includes data volume and data content. The data content includes but is not limited to log files, media data and document data. The source identification includes real-time interaction, online question and answer and offline analysis. The real-time interaction indicates that the multi-source heterogeneous data is data from a high-frequency access interaction source, such as financial transactions and monitoring equipment sensor data. The online question and answer indicates that the multi-source heterogeneous data is data from an online interactive transmission source, such as hospital online diagnosis and social media chat. The offline analysis indicates that the unit heterogeneous data is data from an offline storage and analysis source, such as enterprise database content organization and stored data preprocessing analysis. The multi-source heterogeneous data and source identification are collected in the form of user interactive upload.

[0039] Please continue reading Figure 1 As shown, the computer data processing system based on machine learning also includes:

[0040] The data timestamp construction module is used to number the multi-source heterogeneous data according to the collection time to obtain a timestamp number. The data timestamp construction module is connected to the multi-source data collection module.

[0041] Specifically, the data timestamp construction module in this embodiment numbers the days corresponding to the collection time in chronological order to obtain a day number, and numbers each second in a day in chronological order with a precision of seconds to obtain a second number, and combines the day number and the second number as a timestamp number, and sets the timestamp number to (d, t), where d represents the day number and d∈N + , t represents the second number, t∈N + And t≤86400.

[0042] Specifically, in this embodiment, the data timestamp construction module is used to analyze the sequence number to ensure the timing and continuity of data processing, avoid data conflicts, and improve the standardization of the processing flow.

[0043] Please continue reading Figure 1 As shown, the computer data processing system based on machine learning also includes:

[0044] The data multidimensional classification module is used to classify the security level of the data content of multi-source heterogeneous data, and to construct a three-dimensional classification model based on the real-time nature, data volume and security level of the multi-source heterogeneous data to determine the comprehensive type of the data. The data multidimensional classification module is connected to the data timestamp construction module.

[0045] See also Figure 2 As shown, the data multidimensional classification module includes:

[0046] The security level division unit is used to divide the security level according to the data content of multi-source heterogeneous data.

[0047] Specifically, the security level division unit described in this embodiment divides the security level according to the data content of multi-source heterogeneous data. The security level division unit sets the security level of data content that is private type data to level one, sets the security level of data content that is verified access type data to level two, and sets the security level of data content that is public type data to level three.

[0048] Specifically, the data content in this embodiment is public type data, including but not limited to media information, bidding information and other data open to the public; the data content is verified access type data, including but not limited to server log data and corporate document data, etc., which require user login verification to access; the data content is private type data, including but not limited to user biometric data and other data about the user's personal privacy.

[0049] It is understandable that the present embodiment does not impose any specific limitations on the security level division process, and those skilled in the art can freely set it up, as long as it satisfies the classification of security levels of multi-source heterogeneous data according to different data content types. For example, it can also be set up to divide security levels according to the data transmission method.

[0050] Specifically, in this embodiment, the security level division unit divides the security level according to the data content to ensure that sensitive data is protected first, reduce the risk of data leakage, optimize resource allocation, and avoid excessive processing of low-security level data.

[0051] Please continue reading Figure 2 As shown, the data multidimensional classification module also includes:

[0052] The data volume grading unit is used to divide the data volume levels according to the data volume of multi-source heterogeneous data.

[0053] Specifically, the data volume grading unit in this embodiment compares and analyzes the data volume of multi-source heterogeneous data to classify the data volume levels. If V ≤ v1, the data volume grading unit sets the data volume level of the currently analyzed multi-source heterogeneous data to level one; if v1 < V ≤ v2, the data volume grading unit sets the data volume level of the currently analyzed multi-source heterogeneous data to level two; and if V > v2, the data volume grading unit sets the data volume level of the currently analyzed multi-source heterogeneous data to level three. V represents the data volume of the multi-source heterogeneous data, v1 represents a first data volume threshold, 1 ≤ v1 ≤ 2, and v2 represents a second data volume threshold, 9 ≤ v2 ≤ 10. It will be understood that the data volume threshold values ​​are not specifically limited in this embodiment and can be freely set by those skilled in the art as long as they meet the data volume classification requirements. The optimal data volume threshold values ​​are: v1 = 1, v2 = 10. The data volume and data volume threshold values ​​of the multi-source heterogeneous data are expressed in GB.

[0054] Specifically, in this embodiment, the data volume classification unit is used to classify data into levels according to the size of the data volume, so as to help the system reasonably allocate computing resources, avoid large data volume tasks from occupying too many resources, and improve the overall processing efficiency of the system.

[0055] Please continue reading Figure 2 As shown, the data multidimensional classification module also includes:

[0056] The time sensitivity analysis unit is used to classify delay tolerance levels according to source identification.

[0057] Specifically, the time sensitivity analysis unit in this embodiment divides the delay tolerance level according to the source identifier. The time sensitivity analysis unit sets the delay tolerance level of multi-source heterogeneous data with the source identifier as real-time interaction to level one; sets the delay tolerance level of multi-source heterogeneous data with the source identifier as online question and answer to level two; and sets the delay tolerance level of multi-source heterogeneous data with the source identifier as offline analysis to level three.

[0058] It is understandable that the present embodiment does not impose any specific restrictions on the division of delay tolerance levels, and those skilled in the art can freely set them, as long as they meet the division of the maximum delay for processing data from different sources, and use the delay tolerance level to represent the degree of tolerance for processing delay time.

[0059] Specifically, in this embodiment, the time sensitivity analysis unit divides the delay tolerance level according to the data source identifier to ensure that data with high real-time requirements are processed first, reduce delays, and improve system response speed.

[0060] Please continue reading Figure 2 As shown, the data multidimensional classification module also includes:

[0061] A three-dimensional classification construction unit is used to construct a three-dimensional classification model based on the delay tolerance level, data volume level and security level of multi-source heterogeneous data to determine the comprehensive type of data. The three-dimensional classification construction unit is connected to the security level division unit, data volume classification unit and time sensitivity analysis unit.

[0062] Specifically, the three-dimensional classification construction unit in this embodiment encodes the delay tolerance level, data volume level and security level to obtain a digital level, and quantitatively represents the delay tolerance level, data volume level and security level.

[0063] Specifically, in this embodiment, when encoding the delay tolerance level, data volume level, and security level, 1, 2, and 3 are used to represent level one, level two, and level three, respectively.

[0064] Specifically, the three-dimensional classification construction unit in this embodiment analyzes the type judgment parameter according to the digitization level of the delay tolerance level, data volume level and security level of the multi-source heterogeneous data, sets the type judgment parameter to S, sets S=s1×s2×s3×ln[s1+s2+s3], and compares and analyzes the type judgment parameter to judge the comprehensive type of the data. If S<α1, the three-dimensional classification construction unit determines that the comprehensive type of the data is an emergency security transaction; if α1≤S<α2, the three-dimensional classification construction unit determines that the comprehensive type of the data is an emergency standard transaction; if α2≤S<α3, the three-dimensional classification construction unit determines that the comprehensive type of the data is a conventional standard transaction; if S≥α3. The three-dimensional classification construction unit determines that the comprehensive type of the data is a batch processing transaction; wherein α1 represents the first type threshold, 1.5≤α1≤2, α2 represents the second type threshold, 3≤α2≤10, α3 represents the third type threshold, 12≤α3≤15, α4 represents the fourth type threshold, 20≤α4≤30. It is understandable that the present embodiment does not impose any specific restrictions on the value of the type threshold, and those skilled in the art can freely set it as long as it satisfies the judgment of the comprehensive type of the data. The optimal value of the type threshold is: α1=2, α2=7, α3=15, α4=25.

[0065] Specifically, in this embodiment, the three-dimensional classification construction unit quantifies the delay tolerance level, data volume level and security level through digital coding to construct a three-dimensional classification model, accurately determine the comprehensive type of data, and provide a reliable basis for subsequent priority scoring.

[0066] Please continue reading Figure 1 As shown, the computer data processing system based on machine learning also includes:

[0067] The type feature analysis module is used to perform feature analysis on the data comprehensive type, timestamp number and multi-source heterogeneous data to obtain data features. The type feature analysis module is connected to the data multidimensional classification module.

[0068] Specifically, the type feature analysis module in this embodiment performs feature analysis on the data comprehensive type, timestamp number and multi-source heterogeneous data to obtain data features, sets the data features as L(d), and sets , where H(x) represents the characteristic function, H(x)=sin(x / d)×S×V / [Max(S)×Max(V)], x represents the time variable parameter, x=d×t×2π / 86400, and Max() represents extracting the maximum value of the data in the brackets.

[0069] Specifically, in this embodiment, the type feature analysis module combines data features with timestamps to generate feature vectors, thereby enhancing the machine learning model's ability to parse data attributes and optimizing classification accuracy.

[0070] Please continue reading Figure 1 As shown, the computer data processing system based on machine learning also includes:

[0071] The priority scoring module is used to weight the real-time nature, data volume, security level and data characteristics of multi-source heterogeneous data to obtain a priority score. It is also used to match and analyze the priority score with the comprehensive data type to update the three-dimensional classification model. The priority scoring module is connected to the type feature analysis module.

[0072] See also Figure 3 As shown, the priority scoring module includes:

[0073] The data scoring analysis unit is used to analyze the grade scores based on the delay tolerance level, data volume level and security level of multi-source heterogeneous data.

[0074] Specifically, the data scoring analysis unit described in this embodiment analyzes the level score based on the delay tolerance level, data volume level, and security level of multi-source heterogeneous data, sets the security level score to B1, and sets B1=100-b1×(n-1), sets the data volume level score to B2, and sets B2=100-Min(Max(lg(V+1),0),3) / 3×100, sets the delay tolerance level to B3, and sets B3=100-b2×(n-1), where b1 represents the first score parameter, 40≤b1≤50, b2 represents the second score parameter, 20≤b2≤30, n represents the digital level of the delay tolerance level, data volume level, and security level, n∈{1,2,3}, and Min() represents the minimum value of the data in the brackets. It can be understood that the values ​​of the score parameters are not specifically limited in this embodiment, and those skilled in the art can freely set them as long as they meet the analysis of the level score. The optimal values ​​of the score parameters are: b1=50, b2=30.

[0075] Specifically, in this embodiment, the data scoring analysis unit generates a grade score based on the delay tolerance level, data volume level and security level, providing a quantitative basis for priority scoring and ensuring that the scoring process is scientific and reasonable.

[0076] Please continue reading Figure 3 As shown, the priority scoring module also includes:

[0077] The scoring weighted analysis unit is used to perform weighted analysis on the grade score and the data features to obtain a priority score. The scoring weighted analysis unit is connected to the data scoring analysis unit.

[0078] Specifically, the score weighted analysis unit described in this embodiment performs a weighted analysis on the grade score and data features to obtain a priority score, and sets the priority score to Z. If L(d)>θ, set Z=β1×B1+β2×B2+β3×B3; otherwise, set Z=[β1+L(d) / 10]×B1+[β2-L(d) / 5]×B2+[β3+L(d) / 10]×B3, where β1 represents the first weight, β2 represents the second weight, and β3 represents the third weight. β1+β2+β3=1 and β1×β2×β3≠0, and θ represents the feature threshold, 0.5≤θ≤0.7. It will be understood that the value of the feature threshold is not specifically limited in this embodiment, and those skilled in the art can freely set it as long as it satisfies the analysis of the priority score. The optimal value of the feature threshold is: θ=0.6.

[0079] Specifically, in this embodiment, the initial values ​​of the weights are set to β1=0.3, β2=0.3, and β3=0.4.

[0080] Specifically, in this embodiment, the score weighted analysis unit combines data features with grade scores to perform weighted analysis to dynamically generate priority scores, thereby ensuring that high-priority tasks are processed in a timely manner and improving the overall efficiency of the system.

[0081] Please continue reading Figure 3 As shown, the priority scoring module also includes:

[0082] The classification matching update unit is used to match and analyze the priority score with the data comprehensive type to update the three-dimensional classification model. The classification matching update unit is connected to the score weighted analysis unit.

[0083] Specifically, the classification matching update unit in this embodiment performs a matching analysis on the priority score and the data comprehensive type, extracts the priority score corresponding to the different data comprehensive types as the comprehensive type score, and arranges the comprehensive type scores corresponding to each data comprehensive type in order from small to large, analyzes the type discrete parameter according to the arranged comprehensive type score, sets the type discrete parameter to R(m), and sets R(m)=(Z 0.75×NU(m) -Z 0.25×NU(m) ) / Min(Max(Z∈U(m)),Min(Z∈U(m+1))), where m represents the data comprehensive type parameter, m∈{1,2,3,4}, m=1 represents the data comprehensive type is an emergency security transaction, m=2 represents the data comprehensive type is an emergency standard transaction, m=3 represents a regular standard transaction, and m=4 represents a batch processing transaction. U(m) represents the set of comprehensive type scores corresponding to each comprehensive data type, NU(m) represents the number of comprehensive type scores corresponding to each comprehensive data type, and Z 0.75×NU(m) It represents the 75% comprehensive type score in the arranged comprehensive type score, Z 0.25×NU(m) It represents the 25% comprehensive type score among the arranged comprehensive type scores.

[0084] Specifically, in this embodiment, when analyzing the discrete type parameters, if m=4, the comprehensive type score in the set U(5) is equal to 0.

[0085] Specifically, the classification matching update unit in this embodiment compares and analyzes the discrete type parameters to update the three-dimensional classification model. If R(m)≤r, the classification matching update unit does not update the three-dimensional classification model. If R(m)>r, the classification matching update unit updates the data comprehensive type corresponding to the data comprehensive type parameter to satisfy S' and Z≤Z 0.5×NU(m) +Min(Z∈U(m)) multi-source heterogeneous data adds priority tags, where S' represents the original comparison analysis process, Z 0.5×NU(m)represents the 50th percentile comprehensive type score among the arranged comprehensive type scores, and r represents the updated matching threshold, where 0.5≤r≤0.6. It is understood that the value of the updated matching threshold is not specifically limited in this embodiment, and those skilled in the art may freely set it, as long as it satisfies the requirements for updating the three-dimensional classification model. The optimal value of the updated matching threshold is: r = 0.55.

[0086] Specifically, in this embodiment, when updating the comparison and analysis process of the data comprehensive type, if the data comprehensive type is an emergency safety matter, the updated comparison process is to retain the original comparison process and for S<α1, and Z≤Z 0.5×NU(1) +Min(Z∈U(1)) adds a priority tag to the multi-source heterogeneous data; if the data integration type is an urgent standard transaction, the updated comparison process is to retain the original comparison process and for α1≤S<α2 and Z≤Z 0.5×NU(2) +Min(Z∈U(2)) adds priority marking to multi-source heterogeneous data.

[0087] Specifically, in this embodiment, the classification matching update unit matches the priority score with the data comprehensive type to dynamically update the three-dimensional classification model, ensuring that the system can adjust the processing strategy according to real-time data changes and improve the system's adaptability.

[0088] Please continue reading Figure 1 As shown, the computer data processing system based on machine learning also includes:

[0089] The multi-source data processing module is used to respond and process multi-source heterogeneous data according to the data comprehensive type and priority scoring order. The multi-source data processing module is connected to the priority scoring module.

[0090] Specifically, the multi-source data processing module in this embodiment responds to and processes multi-source heterogeneous data according to the data integration type and priority score order. The multi-source heterogeneous data processing module sorts the multi-source heterogeneous data with the same data integration type and with priority tags and without priority tags in descending order according to the priority score, and preferentially processes the multi-source heterogeneous data with priority tags in the order of arrangement, and then processes the multi-source heterogeneous data without priority tags in the order of arrangement. If Z>z1, the multi-source data processing module allocates the multi-source heterogeneous data to the GPU acceleration queue for processing; if Z<z2, the multi-source data processing module classifies the multi-source heterogeneous data into the night batch task pool for processing; if z2≤Z≤z1, the multi-source data processing module processes the multi-source heterogeneous data normally; wherein, z1 represents the first processing threshold, 90≤z1≤95, and z2 represents the second processing threshold, 40≤z2≤60. It is understandable that the present embodiment does not impose any specific restrictions on the value of the processing threshold, and those skilled in the art can freely set it as long as it satisfies the processing of multi-source heterogeneous data. The optimal values ​​of the processing threshold are: z1=92, z2=50.

[0091] Please continue reading Figure 1 As shown, the computer data processing system based on machine learning also includes:

[0092] The processing anomaly monitoring module is used to count the number of anomaly processing of multi-source heterogeneous data and compare and analyze the number of anomaly processing to trigger the rule optimization of the three-dimensional classification model. The processing anomaly monitoring module is connected to the multi-source data processing module.

[0093] See also Figure 4 As shown, the abnormality monitoring module includes:

[0094] A processing time acquisition unit is used to acquire the processing time of multi-source heterogeneous data. The processing time of multi-source heterogeneous data is the time interval between the application time and the processing completion time of multi-source heterogeneous data. The acquisition method is to acquire the data through the system operation management platform data import;

[0095] The timeout processing judgment unit is used to judge the completion type of the multi-source heterogeneous data according to the processing time and the delay tolerance level. The timeout processing judgment unit is connected to the processing time acquisition unit.

[0096] Specifically, the timeout processing judgment unit in this embodiment judges the completion type of the multi-source heterogeneous data according to the processing time, the timestamp number and the delay tolerance level. When the delay tolerance level is level one, if T≤γ1, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is normal completion; if T>γ1, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is timeout completion. If the processing time is a missing value, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is incomplete. When the delay tolerance level is level two, if T≤γ2, The timeout processing judgment unit determines the completion type of the multi-source heterogeneous data as normal completion. If T>γ2, the timeout processing judgment unit determines the completion type of the multi-source heterogeneous data as timeout completion. If the processing duration is missing, the timeout processing judgment unit determines the completion type of the multi-source heterogeneous data as incomplete. When the delay tolerance level is level 3, the timeout processing judgment unit does not judge the completion type. Wherein, T represents the processing duration, γ1 represents the first delay threshold, 0.05≤γ1≤0.07, and γ2 represents the second delay threshold, 1.5≤γ2≤2.2. It is understood that the values ​​of the delay thresholds are not specifically limited in this embodiment and can be freely set by those skilled in the art as long as they meet the requirements for the completion type. The optimal values ​​of the delay thresholds are γ1=0.05 and γ2=2. The units of the delay thresholds and processing duration are seconds.

[0097] Specifically, in this embodiment, the timeout processing judgment unit judges the data completion type according to the processing duration and the delay tolerance level to ensure that the system can identify timed-out or unfinished tasks and provide a basis for exception handling.

[0098] Please continue reading Figure 4 As shown, the abnormality monitoring module further includes:

[0099] The exception processing analysis unit is used to count the number of exception processing times according to the completion type of multi-source heterogeneous data, and compare and analyze the number of exception processing times to trigger the rule optimization of the three-dimensional classification model. The exception processing analysis unit is connected to the timeout processing judgment unit.

[0100] Specifically, the exception handling analysis unit in this embodiment counts the number of completion types of timeout completion as the number of timeout processing times, counts the number of completion types of incomplete as the number of incomplete times, and takes the sum of 70% of the timeout processing times and the incomplete times as the number of exception handling times.

[0101] Specifically, the exception handling analysis unit in this embodiment triggers the rule optimization of the three-dimensional classification model when the number of timeout processing times of multi-source heterogeneous data of the same data comprehensive type in one second is greater than or equal to k times. The rule optimization of the three-dimensional classification model is to optimize the weighted analysis process of the grade score according to the number of timeout processing times of multi-source heterogeneous data of the same data comprehensive type in one second, optimize the weight of each grade score, and logarithmically correlate the weight of each grade score with the number of timeout processing times to optimize the three-dimensional classification model. The first weight after optimization is β1', and β1'=β1-lnK is set. k / 2, the optimized second weight is β2', set β2'=β2-lnK k / 2, the optimized third weight is β3', set β3'=β3+lnK k k represents the processing count threshold, 3≤k≤5, and K represents the number of timeouts for processing multi-source heterogeneous data of the same data type in one second. It is understood that this embodiment does not impose a specific limit on the value of the processing count threshold, and those skilled in the art may freely set it, as long as it triggers the optimization of the rules of the three-dimensional classification model. The optimal value of the processing count threshold is: k=3.

[0102] Specifically, in this embodiment, the exception handling analysis unit counts the number of exception handling times and triggers rule optimization, dynamically adjusts the weight distribution of the classification model, improves the system's ability to handle abnormal situations, and ensures long-term stable operation of the system.

[0103] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not limitations on the implementation methods of the present invention. For ordinary technicians in this field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A computer data processing system based on machine learning, characterized in that: include: The data multi-dimensional classification module is used to classify the security level of the data content of multi-source heterogeneous data and build a three-dimensional classification model based on the real-time nature, data volume and security level of multi-source heterogeneous data to determine the comprehensive type of data; Type feature analysis module, used to perform feature analysis on data comprehensive type, timestamp number and multi-source heterogeneous data to obtain data features; The priority scoring module is used to weight the real-time performance, data volume, security level, and data characteristics of multi-source heterogeneous data to obtain a priority score. It is also used to match and analyze the priority score with the comprehensive data type to update the three-dimensional classification model. The data multi-dimensional classification module is provided with a security level division unit, which is used to divide the security level of the multi-source heterogeneous data according to the data content. The security level division unit divides the security level of the multi-source heterogeneous data into three levels according to the different data contents; The data multi-dimensional classification module is further provided with a data volume grading unit, which is used to compare and analyze the data volume of multi-source heterogeneous data to divide the data volume level. The data volume grading unit divides the multi-source heterogeneous data into three levels according to the size of the data volume; The data multi-dimensional classification module is further provided with a time sensitivity analysis unit, which is used to classify delay tolerance levels according to source identification. The time sensitivity analysis unit classifies the delay tolerance levels of multi-source heterogeneous data into three levels according to different source identifications.

2. The computer data processing system based on machine learning according to claim 1, characterized in that The system further comprises: Multi-source data acquisition module, used to collect multi-source heterogeneous data and source identification in real time; The data timestamp construction module is used to number multi-source heterogeneous data according to the collection time to obtain the timestamp number; Multi-source data processing module, used to respond and process multi-source heterogeneous data according to the data comprehensive type and priority score order; The anomaly processing monitoring module is used to count the number of anomaly processing times of multi-source heterogeneous data and perform comparative analysis on the anomaly processing times to trigger rule optimization of the three-dimensional classification model.

3. The computer data processing system based on machine learning according to claim 1, characterized in that The data multi-dimensional classification module is further provided with a three-dimensional classification construction unit, which is used to encode the delay tolerance level, data volume level and security level to obtain a digital level, and quantitatively represent the delay tolerance level, data volume level and security level; The three-dimensional classification construction unit analyzes the type judgment parameters according to the digital level of the delay tolerance level, data volume level and security level of the multi-source heterogeneous data to obtain the type judgment parameter S, and compares and analyzes the type judgment parameters to judge the data synthesis type, and divides the data synthesis type of the multi-source heterogeneous data into four categories according to the size of the type judgment parameters. The data synthesis types include emergency security affairs, emergency standard affairs, conventional standard affairs and batch processing affairs.

4. The computer data processing system based on machine learning according to claim 3, characterized in that The type feature analysis module performs feature analysis on the data comprehensive type, timestamp number and multi-source heterogeneous data to obtain the data feature L(d).

5. The computer data processing system based on machine learning according to claim 4, characterized in that The priority scoring module is provided with a data scoring analysis unit, which is used to analyze the grade scores according to the delay tolerance level, data volume level and security level of multi-source heterogeneous data to obtain a security level score B1, a data volume level score B2 and a delay tolerance level B3; The priority scoring module is also provided with a scoring weighted analysis unit, which is used to perform weighted analysis on the grade score and data features to obtain a priority score. The priority score is set to Z. If L(d)>θ, set Z=β1×B1+β2×B2+β3×B3; otherwise, set Z=[β1+L(d) / 10]×B1+[β2-L(d) / 5]×B2+[β3+L(d) / 10]×B3, wherein β1 represents the first weight, β2 represents the second weight, β3 represents the third weight, β1+β2+β3=1 and β1×β2×β3≠0.

6. The computer data processing system based on machine learning according to claim 5, characterized in that The priority scoring module is further provided with a classification matching update unit, which is used to perform matching analysis on the priority score and the data comprehensive type, extract the priority score corresponding to the different data comprehensive types as the comprehensive type score, and arrange the comprehensive type scores corresponding to each data comprehensive type in order from small to large, and analyze the type discrete parameter according to the arranged comprehensive type score to obtain the type discrete parameter R(m); The classification matching update unit compares and analyzes the discrete parameters of the type to update the three-dimensional classification model. When R(m)>r, the classification matching update unit matches the discrete parameters of the type to update the three-dimensional classification model. 0.5×NU(m) +Min(Z∈U(m)) adds a priority tag to the multi-source heterogeneous data, r represents the updated matching threshold, U(m) represents the set of comprehensive type scores corresponding to each comprehensive data type, and NU(m) represents the number of comprehensive type scores corresponding to each comprehensive data.

7. The computer data processing system based on machine learning according to claim 2, characterized in that The multi-source data processing module responds to and processes multi-source heterogeneous data according to the data integration type and priority score order. The multi-source data processing module sorts the multi-source heterogeneous data with the same data integration type and with priority tags and without priority tags in descending order according to the priority score, and preferentially processes the multi-source heterogeneous data with priority tags in the order of arrangement, and then processes the multi-source heterogeneous data without priority tags in the order of arrangement. If Z>z1, the multi-source data processing module allocates the multi-source heterogeneous data to the GPU acceleration queue for processing; if Z<z2, the multi-source data processing module classifies the multi-source heterogeneous data into the night batch task pool for processing; if z2≤Z≤z1, the multi-source data processing module processes the multi-source heterogeneous data normally; wherein z1 represents a first processing threshold, z2 represents a second processing threshold, and Z represents a priority score.

8. The computer data processing system based on machine learning according to claim 7, characterized in that The processing abnormality monitoring module is provided with a timeout processing judgment unit, which is used to judge the completion type of the multi-source heterogeneous data according to the processing time, the timestamp number and the delay tolerance level. When the delay tolerance level is level one, if T≤γ1, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is normal completion; if T>γ1, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is timeout completion; if the processing time is a missing value, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is incomplete; when the delay tolerance level is level two, if T≤γ2, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is normal completion; if T>γ2, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is timeout completion; if the processing time is a missing value, the timeout processing judgment unit judges that the completion type of the multi-source heterogeneous data is incomplete; when the delay tolerance level is level three, the timeout processing judgment unit does not judge the completion type; wherein T represents the processing time, γ1 represents the first delay threshold, and γ2 represents the second delay threshold.

9. The computer data processing system based on machine learning according to claim 8, characterized in that The processing exception monitoring module is further provided with an exception processing analysis unit, which is used to count the number of completion types of timeout completion as the number of timeout processing times, count the number of completion types of incomplete as the number of incomplete times, and take the sum of 70% of the timeout processing times and the incomplete times as the number of exception processing times; When the number of timeout processing times of multi-source heterogeneous data of the same data integration type in one second is greater than or equal to k times, the exception processing analysis unit triggers the rule optimization of the three-dimensional classification model. The rule optimization of the three-dimensional classification model is to optimize the weighted analysis process of the grade score according to the number of timeout processing times of multi-source heterogeneous data of the same data integration type in one second, optimize the weight of each grade score, and logarithmically correlate the weight of each grade score with the number of timeout processing times to optimize the three-dimensional classification model, where k represents the processing number threshold.

Images