Cloud-based data service and processing optimization system

Abstract

This invention relates to the field of data optimization and processing technology, specifically to a cloud platform-based data service and processing optimization system, comprising: a data quality assessment module: performing real-time quality assessment on the input data stream, generating quality assessment indicators including integrity score, noise density, and semantic consistency, and integrating these indicators to generate a window-level comprehensive credibility score; a processing strategy generation module: dividing the data stream into high-credibility core data and low-credibility edge data based on the comprehensive credibility score; and a cross-channel resource collaborative scheduling module: deploying parallel computing resources with fixed time windows in deterministic computing channels and enabling dynamically scalable fuzzy computing unit allocation in probabilistic processing channels. This invention effectively avoids low-quality data interfering with high-priority computing tasks while ensuring accurate processing of core data, thereby improving the data credibility and computational stability of the cloud computing platform.

Description

Technical Field

[0001] This invention relates to the field of data optimization and processing technology, and in particular to a data service and processing optimization system based on a cloud platform. Background Technology

[0002] With the rapid development of cloud computing and big data technologies, cloud platforms have become an important infrastructure for data storage, processing and analysis. In various application scenarios, cloud platforms need to efficiently manage and schedule large amounts of heterogeneous data to meet business needs. However, existing data processing systems still face technical bottlenecks when dealing with complex data environments.

[0003] In modern cloud computing environments, data sources are complex, including sensor data, log data, and business transaction data. These data often have problems such as missing data, inconsistencies, and noise interference. If an effective data quality assessment is not conducted before computation, low-quality data may enter the computation process, leading to deviations in the computation results and affecting the accuracy of decision-making. Traditional data quality management methods mainly rely on static rule definitions, which are difficult to adapt to large-scale, dynamically changing data flow environments and cannot achieve accurate measurement and adaptive optimization of data quality.

[0004] Existing cloud platforms often employ fixed resource allocation models for data computing, meaning they allocate computing resources to different tasks according to preset strategies. However, in practical applications, due to the volatility of data quality, fixed resource allocation often leads to two extreme situations: first, high-quality and low-quality data are mixed and processed, causing computing resources to be consumed by low-quality data, thus reducing overall computing efficiency; second, computing resources fail to dynamically match data demands, resulting in resource waste or computing bottlenecks. Traditional cloud computing architectures lack the ability to perceive data quality and cannot adaptively adjust computing strategies based on data quality, thereby affecting the utilization rate and processing performance of computing resources. Summary of the Invention

[0005] This invention provides a data service and processing optimization system based on a cloud platform.

[0006] A cloud-based data service and processing optimization system includes:

[0007] Data quality assessment module: performs real-time quality assessment on the input data stream, generates quality assessment indicators including integrity score, noise density and semantic consistency, and integrates the various quality assessment indicators to generate a window-level comprehensive credibility score;

[0008] Processing strategy generation module: Based on the comprehensive credibility score, the data stream is divided into high-credibility core data and low-credibility edge data, a deterministic computing channel is allocated to the core data, and a probabilistic processing channel is constructed for the edge data;

[0009] Cross-channel resource collaborative scheduling module: Deploys parallel computing resources with fixed time windows in the deterministic computing channel, and enables dynamically scalable fuzzy computing unit allocation in the probabilistic processing channel. The two channels share a cache pool and dynamically migrate computing tasks according to changes in data quality.

[0010] Optionally, the data quality assessment module specifically includes:

[0011] Multi-dimensional computation unit: The input data stream is sliced ​​into time windows, and the following is executed synchronously within each time window:

[0012] a. Integrity score calculation: Calculate the missing rate of required fields and generate a dynamic integrity score by combining the historical reliability weight of the data source;

[0013] b. Noise density detection: For numerical data, the moving standard deviation algorithm is used to identify abnormal fluctuations; for text data, the noise density is calculated by the proportion of irrelevant characters.

[0014] c. Semantic consistency verification: The frequency of semantic conflicts between data content and preset business rules is detected through a pre-trained business knowledge graph matching unit.

[0015] The comprehensive scoring unit integrates the results of integrity score, noise density, and semantic consistency to obtain a comprehensive credibility score.

[0016] Optionally, the business knowledge graph matching unit includes:

[0017] Knowledge graph storage sub-unit: Stores the semantic network related to business rules, including entities, relations, and rule sets;

[0018] Semantic parsing and vectorization subunit: Transforms data records into vector representations that can be compared and matched with knowledge graphs;

[0019] Rule matching engine: Based on graph traversal, it detects whether data records violate business rules;

[0020] Conflict frequency calculation subunit: counts the number of semantic conflicts that occur and outputs the semantic conflict frequency.

[0021] The specific matching method is as follows:

[0022] First, semantic analysis is performed on the input data, and it is transformed into a structured vector representation that can be used to match business rules.

[0023] Extract rule subgraphs from the knowledge graph that are relevant to the current data, which contain entities (customers, transactions) and their relationships (transfer limits, credit ratings);

[0024] Calculate the degree of matching between data content and knowledge graph rules, and use vector similarity method to determine whether the data conforms to the established business rules;

[0025] If the number of times the data violates the rules is counted and the matching degree is lower than the set threshold, then the data is determined to have a semantic conflict.

[0026] Optionally, the processing strategy generation module includes a threshold division unit: when the comprehensive credibility score... When marked as high-confidence core data, it is allocated to the deterministic computing channel; when At that time, it is marked as low-confidence edge data and assigned to the probabilistic processing channel.

[0027] Optionally, the overall credibility score is normalized to [0,1], when At that time, one of the following operations will be automatically triggered based on the real-time pressure of the data stream:

[0028] a. If the current deterministic channel load rate L core <η safe ,promote To the core data, η safe Take 70%;

[0029] b. Otherwise, downgrade to edge data and inject probabilistic channels.

[0030] High confidence threshold θ high The value is 0.6, which is the low confidence threshold θ. low The value is 0.4.

[0031] Optionally, the cross-channel resource collaborative scheduling module specifically includes:

[0032] Fixed time window deployment unit: Based on the comprehensive credibility score of core data, the length of the computation time window for that data is dynamically adjusted. The calculation of the time window length takes into account the relationship between the comprehensive credibility score and the historical average score, so that high-credibility data can obtain a longer processing time. A linear mapping model is adopted to make the allocation of parallel computing resources proportional to the comprehensive credibility score of each data time window. Based on the sum of the comprehensive credibility scores of all core data time windows, the proportion of the comprehensive credibility score of a single time window is calculated and then multiplied by the total available computing resources to obtain the computing resources that can be allocated for that time window.

[0033] Fuzzy computing unit: For edge data, the number of fuzzy computing units is determined by the noise density in the current time window. The actual number of fuzzy computing units required is calculated based on the noise density of text data, the noise figure of numerical data, the noise density tolerance coefficient, and the maximum number of fuzzy computing units.

[0034] Cross-channel task migration unit: includes migration from deterministic computing channels to probabilistic processing channels and migration from probabilistic processing channels to deterministic computing channels.

[0035] Optionally, the calculation accuracy of the fuzzy computing unit is adaptively adjusted according to the frequency of semantic conflicts. When the frequency of semantic conflicts is high, the calculation accuracy decreases accordingly to reduce computational overhead; when the frequency of semantic conflicts is low, the calculation accuracy is increased.

[0036] Optionally, the conditions for migrating from the deterministic computation channel to the probabilistic processing channel include:

[0037] When the processing time of the core data time window exceeds its allocated time window and its comprehensive credibility score drops below the credibility threshold, the data in that time window is downgraded to the probabilistic processing channel to release core computing resources. The amount of computing resources migrated is calculated by multiplying the original computing resources of the time window by the ratio of its current comprehensive credibility score to the high credibility threshold.

[0038] The conditions for migrating from the probabilistic processing channel to the deterministic computation channel include:

[0039] For edge data, if the updated comprehensive credibility score exceeds the adjustment range of the high credibility threshold after fuzzy computation, it is migrated to the deterministic computation channel and allocated computing resources. The allocation of computing resources is proportional to the comprehensive credibility score, so that edge data with high credibility can obtain more computing resources.

[0040] Optionally, the linear mapping model is represented as:

[0041] Where K is the total number of current core data time windows, and R total This represents the total amount of available computing resources. To allocate computing resources to the k-th core data window, The overall credibility score for the i-th time window is given. The overall credibility score for the j-th time window.

[0042] Optionally, the number of fuzzy computing units U fuzzy Binding to noise density:

[0043] in, For noise density in text data, For numerical data, ρ represents the noise figure. noise U is the noise density tolerance factor. max This represents the maximum number of fuzzy computational units.

[0044] The beneficial effects of this invention are:

[0045] This invention utilizes a data quality assessment module to perform real-time quality analysis on the input data stream. It employs a three-dimensional comprehensive evaluation based on integrity scoring, noise density, and semantic consistency to generate a window-level comprehensive credibility score. This scoring system accurately identifies data quality differences and, combined with a dynamic dual-threshold partitioning strategy, divides the data stream into high-credibility core data and low-credibility edge data in real time. Through this mechanism, the system effectively avoids low-quality data interfering with high-priority computing tasks while ensuring accurate processing of core data, thereby improving the data credibility and computing stability of the cloud computing platform.

[0046] This invention enables intelligent collaboration between deterministic computing channels and probabilistic processing channels through a cross-channel resource collaborative scheduling module. The core data time window employs a dynamically adjusted fixed-time-window processing strategy, automatically expanding or contracting the time window based on credibility scores to adapt to the computational needs of data with varying credibility levels. Simultaneously, a credibility-resource linear mapping model is used to ensure that the allocation of computing resources matches data quality, thereby optimizing the computational priority of high-credibility data. For edge data, a noise density-driven fuzzy computing unit dynamic scaling mechanism is employed, enabling data streams with higher noise levels to obtain adaptive computing resources. Furthermore, the computational accuracy is dynamically adjusted through semantic conflict frequency feedback, thereby reducing computational costs and improving the utilization rate of computing resources.

[0047] This invention achieves intelligent migration of cross-channel tasks through a shared cache pool, effectively improving the load balancing of computing resources. When the core computing channel experiences excessive load due to data quality degradation or processing timeouts, the system automatically triggers data degradation, migrating low-quality data from the core channel to a probabilistic processing channel to free up high-priority computing resources. Simultaneously, when edge data, after fuzzy computation, achieves a credibility score within the high-credibility threshold adjustment range, it is automatically upgraded to the core channel, and computing resources are allocated according to the credibility ratio. This bidirectional migration mechanism ensures that the system can dynamically adjust the assignment of computing tasks based on data quality, improving the accuracy of data computation and optimizing the overall allocation and scheduling capabilities of computing resources. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of the system functional modules according to an embodiment of the present invention;

[0050] Figure 2This is a schematic diagram of the cross-channel resource collaborative scheduling module according to an embodiment of the present invention. Detailed Implementation

[0051] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should also be noted that, to make the embodiments more comprehensive, the following embodiments are the best and preferred embodiments, and those skilled in the art can use other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0052] It should be noted that the use of terms such as "an embodiment," "an embodiment," "an exemplary embodiment," and "some embodiments" in the specification indicates that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the art.

[0053] In general, terms can be understood, at least in part, from their use in context. For example, depending at least in part on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending at least in part on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0054] like Figure 1-Figure 2 As shown, the cloud platform-based data service and processing optimization system includes:

[0055] Data quality assessment module: performs real-time quality assessment on the input data stream, generates quality assessment indicators including integrity score, noise density and semantic consistency, and integrates the various quality assessment indicators to generate a window-level comprehensive credibility score;

[0056] Processing strategy generation module: Based on the comprehensive credibility score, the data stream is divided into high-credibility core data and low-credibility edge data. A deterministic computing channel is allocated to the core data, and a probabilistic processing channel is constructed for the edge data.

[0057] Cross-channel resource collaborative scheduling module: Deploys parallel computing resources with fixed time windows in the deterministic computing channel, and enables dynamically scalable fuzzy computing unit allocation in the probabilistic processing channel. The two channels share a cache pool and dynamically migrate computing tasks according to changes in data quality.

[0058] The data quality assessment module specifically includes:

[0059] 1. Divide the input data stream into time windows of length T seconds, and perform operations synchronously within each window:

[0060] 1.1 Integrity Score Calculation: For N data records within the i-th time window, calculate the dynamic integrity score. in, K represents the total number of missing required fields in this window. req The number of required fields is determined by preset business rules. The historical reliability weight of the data source is calculated as follows: H represents the number of historical windows. This represents the number of error records for this data source within the `it` window;

[0061] 1.2. Noise density detection: Calculate the moving standard deviation noise figure from the logarithmic data.

[0062] Where L is the total number of numeric fields in the current window, σ s Let be the sliding standard deviation of the data for the s-th field within the window (the sliding window size is 5 rows), and s be the moving average of the s-th field.

[0063] Calculate the density of irrelevant character noise for text data. in, Let be the number of characters in the j-th record that are not part of the preset character set (excluding printable ASCII characters). The total number of characters in the j-th record;

[0064] 1.3. Semantic Consistency Verification: Calculate the frequency of semantic conflicts through knowledge graph matching.

[0065] Where R is the total number of preset business rules, KG r Let r be a subgraph of the knowledge graph of the r-th business rule. It is the j-th data record within the i-th time window. Indicates when data record With KG r The value is 1 if there is a semantic conflict, and 0 otherwise.

[0066] 2. Comprehensive Scoring Unit: The comprehensive credibility score is obtained by fusing the results of integrity score, noise density, and semantic consistency.

[0067] Where α, β, γ are dynamic weighting coefficients, satisfying α + β + γ = 1, α is 0.4, β is 0.3, and γ is 0.3.

[0068] The business knowledge graph matching unit includes:

[0069] Knowledge graph storage sub-unit: Stores the semantic network related to business rules, including entities, relations, and rule sets;

[0070] Semantic parsing and vectorization subunit: Transforms data records into vector representations that can be compared and matched with knowledge graphs;

[0071] Rule matching engine: Based on graph traversal, it detects whether data records violate business rules;

[0072] Conflict frequency calculation subunit: counts the number of semantic conflicts that occur and outputs the semantic conflict frequency.

[0073] The specific matching method is as follows:

[0074] First, semantic analysis is performed on the input data, and it is transformed into a structured vector representation that can be used to match business rules.

[0075] Extract rule subgraphs from the knowledge graph that are relevant to the current data, which contain entities (customers, transactions) and their relationships (transfer limits, credit ratings);

[0076] Calculate the degree of matching between data content and knowledge graph rules, and use vector similarity method to determine whether the data conforms to the established business rules;

[0077] If the number of times the data violates the rules is counted and the matching degree is lower than the set threshold, then the data is determined to have a semantic conflict.

[0078] The processing strategy generation module includes a threshold division unit: when the overall credibility score is... When marked as high-confidence core data, it is allocated to the deterministic computing channel; when At that time, it is marked as low-confidence edge data and assigned to the probabilistic processing channel.

[0079] The overall credibility score is normalized to [0,1], when At that time, one of the following operations will be automatically triggered based on the real-time pressure of the data stream:

[0080] a. If the current deterministic channel load rate L core <η safe ,promote To the core data, η safe Take 70%;

[0081] b. Otherwise, downgrade to edge data and inject probabilistic channels.

[0082] High confidence threshold θ highThe value is 0.6, which is the low confidence threshold θ. low The value is 0.4.

[0083] The cross-channel resource collaborative scheduling module specifically includes:

[0084] 1. Fixed time window deployment for deterministic computation channels:

[0085] Time window length T core Comprehensive credibility score of core data Positive correlation:

[0086] Among them, T base As the reference time window, μ Q The historical reliability score average is used to balance the dynamic adjustment of the calculation window length;

[0087] Parallel computing resource allocation adopts a reliability-resource linear mapping model:

[0088] Where K is the total number of current core data windows, and R total This represents the total amount of available computing resources. The computing resources allocated to the k-th core data window;

[0089] 2. Dynamic scaling mechanism for probabilistic processing channels:

[0090] Number of fuzzy computing units U fuzzy Dynamic binding with noise density:

[0091] in, For noise density in text data, For numerical data, ρ represents the noise figure. noise U is the noise density tolerance factor. max This represents the maximum number of fuzzy computational units.

[0092] Fuzzy calculation accuracy P fuzzy Adaptive adjustment based on semantic conflict frequency:

[0093] in, Let λ represent the semantic conflict frequency within the i-th window, N be the total number of data records within the window, and λ be the number of records within the window. conflict This is the collision frequency attenuation factor (preset value).

[0094] 3. Cross-channel task migration via shared cache pool:

[0095] Migrating from core channels to edge channels: When the core data window processing times out (t process >T core And credibility Descending to θ high When -2Δ, the current window data is downgraded to the probabilistic channel, θ high -2Δ is the credibility threshold, which releases computing resources, expressed as:

[0096] Among them, t process R represents the data processing time for the current window. release This indicates the amount of computing resources released from the core channel;

[0097] Migration from edge channels to core channels: When edge data meets the following conditions after fuzzy computation:

[0098]

[0099] Immediately migrate it to the core channel and allocate initial resources:

[0100] Where Δ is the upgrade buffer threshold (to prevent frequent switching of edge data), and R avg R represents the current average resource usage of the core channel. migrate The final step is to migrate computing resources to the core channel.

[0101] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0102] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A data service and processing optimization system based on a cloud platform, characterized in that: include: Data quality assessment module: performs real-time quality assessment on the input data stream, generates quality assessment indicators including integrity score, noise density and semantic consistency, and integrates the various quality assessment indicators to generate a window-level comprehensive credibility score; Processing strategy generation module: Based on the comprehensive credibility score, the data stream is divided into high-credibility core data and low-credibility edge data, a deterministic computing channel is allocated to the core data, and a probabilistic processing channel is constructed for the edge data; Cross-channel resource collaborative scheduling module: Deploys parallel computing resources with fixed time windows in the deterministic computing channel, and enables dynamically scalable fuzzy computing unit allocation in the probabilistic processing channel. The two channels share a cache pool and dynamically migrate computing tasks according to changes in data quality.

2. The cloud platform-based data service and processing optimization system according to claim 1, characterized in that, The data quality assessment module specifically includes: Multi-dimensional computation unit: The input data stream is sliced ​​into time windows, and the following is executed synchronously within each time window: a. Integrity score calculation: Calculate the missing rate of required fields and generate a dynamic integrity score by combining the historical reliability weight of the data source; b. Noise density detection: For numerical data, the moving standard deviation algorithm is used to identify abnormal fluctuations; for text data, the noise density is calculated by the proportion of irrelevant characters. c. Semantic consistency verification: The frequency of semantic conflicts between data content and preset business rules is detected through a pre-trained business knowledge graph matching unit. The comprehensive scoring unit integrates the results of integrity score, noise density, and semantic consistency to obtain a comprehensive credibility score.

3. The cloud platform-based data service and processing optimization system according to claim 2, characterized in that, The business knowledge graph matching unit includes: Knowledge graph storage sub-unit: Stores the semantic network related to business rules, including entities, relations, and rule sets; Semantic parsing and vectorization subunit: Transforms data records into vector representations that can be compared and matched with knowledge graphs; Rule matching engine: Based on graph traversal, it detects whether data records violate business rules; Conflict frequency calculation subunit: counts the number of semantic conflicts that occur and outputs the semantic conflict frequency.

4. The cloud platform-based data service and processing optimization system according to claim 1, characterized in that, The processing strategy generation module includes a threshold division unit: when the comprehensive credibility score is... When marked as high-confidence core data, it is allocated to the deterministic computing channel; when At that time, it is marked as low-confidence edge data and assigned to the probabilistic processing channel.

5. The cloud platform-based data service and processing optimization system according to claim 4, characterized in that, The overall credibility score is normalized to [0,1], when At that time, one of the following operations will be automatically triggered based on the real-time pressure of the data stream: a. If the current deterministic channel load rate L core <η safe ,promote To the core data, η safe Take 70%; b. Otherwise, downgrade to edge data and inject probabilistic channels.

6. The cloud platform-based data service and processing optimization system according to claim 2, characterized in that, The cross-channel resource collaborative scheduling module specifically includes: Fixed time window deployment unit: Based on the comprehensive credibility score of core data, the length of the computation time window for that data is dynamically adjusted. The calculation of the time window length takes into account the relationship between the comprehensive credibility score and the historical average score, so that high-credibility data can obtain a longer processing time. A linear mapping model is adopted to make the allocation of parallel computing resources proportional to the comprehensive credibility score of each data time window. Based on the sum of the comprehensive credibility scores of all core data time windows, the proportion of the comprehensive credibility score of a single time window is calculated and then multiplied by the total available computing resources to obtain the computing resources that can be allocated for that time window. Fuzzy computing unit: For edge data, the number of fuzzy computing units is determined by the noise density in the current time window. The actual number of fuzzy computing units required is calculated based on the noise density of text data, the noise figure of numerical data, the noise density tolerance coefficient, and the maximum number of fuzzy computing units. Cross-channel task migration unit: includes migration from deterministic computing channels to probabilistic processing channels and migration from probabilistic processing channels to deterministic computing channels.

7. The cloud platform-based data service and processing optimization system according to claim 6, characterized in that, The calculation accuracy of the fuzzy computing unit is adaptively adjusted according to the frequency of semantic conflicts. When the frequency of semantic conflicts is high, the calculation accuracy decreases accordingly to reduce computational overhead; when the frequency of semantic conflicts is low, the calculation accuracy is increased.

8. The cloud platform-based data service and processing optimization system according to claim 6, characterized in that, The conditions for migrating from the deterministic computation channel to the probabilistic processing channel include: When the processing time of the core data time window exceeds its allocated time window and its comprehensive credibility score drops below the credibility threshold, the data in that time window is downgraded to the probabilistic processing channel to release core computing resources. The amount of computing resources migrated is calculated by multiplying the original computing resources of the time window by the ratio of its current comprehensive credibility score to the high credibility threshold. The conditions for migrating from the probabilistic processing channel to the deterministic computation channel include: For edge data, if the updated comprehensive credibility score exceeds the adjustment range of the high credibility threshold after fuzzy computation, it is migrated to the deterministic computation channel and allocated computing resources. The allocation of computing resources is proportional to the comprehensive credibility score, so that edge data with high credibility can obtain more computing resources.

9. The cloud platform-based data service and processing optimization system according to claim 6, characterized in that, The linear mapping model is expressed as follows: Where K is the total number of current core data time windows, and R total This represents the total amount of available computing resources. To allocate computing resources to the k-th core data window, The overall credibility score for the i-th time window is given. The overall credibility score for the j-th time window.

10. The cloud platform-based data service and processing optimization system according to claim 6, characterized in that, The number of fuzzy computing units U fuzzy Binding to noise density: in, For noise density in text data, For numerical data, ρ represents the noise figure. noise U is the noise density tolerance factor. max This represents the maximum number of fuzzy computational units.

Images