Data management method and system for reducing computing power occupation

Abstract

The invention relates to the technical field of computer data management, and discloses a data management method and system capable of reducing computing power occupancy, and the method comprises the steps: S1, based on data distribution analysis, employing a sparse representation algorithm and a principal component analysis method, carrying out the operation elimination of the redundancy of high-dimensional data features through setting an L1 regularization constraint condition, and obtaining an L1 regularization constraint condition; decomposing the data correlation matrix in combination with a matrix operation mode, extracting feature vectors with top-ranked contribution values, and converting the feature vectors into a sparse low-dimensional matrix; and generating a sparse low-dimensional data matrix. Through combination of sparse representation and principal component analysis, L1 regularization is creatively applied to perform feature screening on high-dimensional data, redundant data items are removed, and principal component features with high correlation are extracted, so that the data dimension is effectively reduced, consumption of computing resources is reduced, and meanwhile, main information features of the data are reserved; the efficiency and accuracy of subsequent processing are improved, and distributed scheduling of tasks and efficient utilization of equipment computing power are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data management, and in particular to a data management method and system for reducing computing power usage. Background Art

[0002] Computer data management is a core technology field in computer science, mainly studying how to efficiently store, organize, process and access data. With the rapid development of technologies such as big data, artificial intelligence and cloud computing, data management technology has gradually evolved from traditional static storage and processing to dynamic, intelligent and distributed management, covering multiple directions such as database management systems, distributed storage, data stream processing, and data compression and encryption. Its goal is to improve data processing efficiency and system computing performance by optimizing data organization structure and access mechanism in the case of rapid growth of data volume, while reducing resource occupation, ensuring data security, reliability and scalability. It is widely used in various high-concurrency and high-performance computing scenarios. Among them, a data management method and system that reduces computing power occupation refers to reducing computing resource consumption in the data processing process by designing efficient data management mechanisms and task allocation strategies, while improving data storage and processing efficiency. This method is suitable for scenarios where computing power allocation needs to be optimized, and can be used in fields such as big data analysis, real-time computing, cloud service optimization and intelligent task scheduling, which helps to reduce energy consumption, save resources and achieve comprehensive improvement of system performance.

[0003] In data management, the redundant characteristics of high-dimensional data are not fully processed in the existing technology. Feature extraction usually relies on fixed dimensionality reduction algorithms, which are difficult to dynamically optimize for different data distributions, resulting in too high data dimensions and increasing the complexity and time consumption of subsequent calculations. In partition management, the partition strategy is mostly static partitioning, which fails to adjust in real time according to changes in access frequency, resulting in extended data access paths and high cross-node communication overhead, affecting data processing efficiency. In event data screening, existing methods often lack efficient filtering mechanisms. They usually compare rules step by step by traversing the entire amount of data. The processing process is lengthy and the computing power overhead is high, and the accuracy is difficult to guarantee. In the task scheduling stage, the allocation method of tasks and devices fails to combine the dynamic changes of computing intensity and device load, which easily leads to uneven resource allocation, resulting in overload or vacancy of some devices, and low overall computing power utilization. These deficiencies make the existing technology show a high waste of resources when facing large-scale data processing or real-time computing needs, and cannot meet the needs of efficient and flexible computing. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention provides a data management method and system for reducing computing power usage, which solves the problem that the redundant characteristics of high-dimensional data are not fully processed in data management, feature extraction usually relies on fixed dimensionality reduction algorithms, and it is difficult to dynamically optimize for different data distributions, resulting in too high data dimensions, increasing the complexity and time consumption of subsequent calculations. In partition management, partition strategies are mostly static partitions, which fail to adjust in real time according to changes in access frequency, resulting in extended data access paths, large cross-node communication overhead, and affecting data processing efficiency.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: A data management method for reducing computing power usage, comprising the following steps: S1: Based on data distribution analysis, sparse representation algorithm and principal component analysis method are used, L1 regularization constraints are set, redundant high-dimensional data features are eliminated, matrix operations are combined to decompose the data correlation matrix, feature vectors with the top 30% contribution rate are extracted, and converted into sparse low-dimensional matrices; sparse low-dimensional data matrices are generated; S2: Based on the sparse low-dimensional data matrix, a dynamic partition optimization algorithm is used to select high-frequency access data sets by building a data access frequency distribution model. The data is allocated to different storage partitions according to weights in combination with the distributed node task table. The partition weights are dynamically adjusted based on real-time monitoring and the allocation results are updated; an optimized dynamic data partition table is generated; S3: Based on the optimized dynamic data partition table, an event-driven filtering algorithm is used to construct a Bloom filter to filter the key event data set that meets the conditions. The data tags are judged item by item according to the defined rule table. The key event identifiers are marked for the data that hits the conditions and then extracted and stored; the key event data stream is generated; S4: Based on the key event data stream, a heterogeneous computing task scheduling algorithm is adopted. By classifying and marking the computing intensity of task feature vectors, light tasks are assigned to the CPU parallel queue for processing and high-intensity tasks are assigned to the GPU batch queue for processing according to the task load analysis results. The distribution of computing tasks of each device is adjusted in combination with the dynamic load model; a set of computing results for efficient processing is generated.

[0006] Preferably, based on data distribution analysis, a sparse representation algorithm and a principal component analysis method are used to eliminate the redundancy of high-dimensional data features by setting L1 regularization constraints, decompose the data correlation matrix in combination with a matrix operation method, extract the feature vectors with the highest contribution value and convert them into a sparse low-dimensional matrix; the specific steps of generating a sparse low-dimensional data matrix are as follows: S101: Based on the high-dimensional data set, the L1 regularization constraint algorithm is used to construct the objective function, take the sparsity of the eigenvalue as the constraint condition, calculate each feature in the data matrix column by column, remove the feature items whose weight value tends to zero under the constraint condition, and generate a preliminary sparse feature matrix; S102: Based on the preliminary sparse feature matrix, a covariance matrix calculation method is used to construct a feature correlation matrix according to the covariance values ​​between features, a feature vector matrix is ​​obtained through feature decomposition, and the feature vectors with the highest weight ranking are extracted as principal components to generate a feature vector matrix; S103: Based on the eigenvector matrix, the high-dimensional data is projected into the principal component space by a matrix projection method, the feature dimension is reduced according to the principal component contribution rate, and the feature dimension whose principal component contribution rate is less than a preset threshold is eliminated to generate a principal component low-dimensional matrix; S104: Based on the principal component low-dimensional matrix, the sparse matrix mapping method is used to optimize the matrix sparsity, retain the sparse features with weights in the top 30% of the principal components, output the final sparse matrix result, and generate a sparse low-dimensional data matrix.

[0007] Preferably, based on the sparse low-dimensional data matrix, a dynamic partition optimization algorithm is adopted to screen the frequently accessed data sets by constructing a data access frequency distribution model, and the data is allocated to different storage partitions according to the weights in combination with the distributed node task table. The partition weights are dynamically adjusted based on real-time monitoring and the allocation results are updated. The specific steps for generating the optimized dynamic data partition table are: S201: constructing a data access frequency model based on a sparse low-dimensional data matrix, counting the number of accesses and time intervals of each data block according to data access log records, calculating the access frequency and generating a data block access frequency ranking, and generating a data access frequency model; S202: Based on the data access frequency model, a K-means clustering algorithm is used to allocate data blocks with access frequencies in the top 30% to priority partitions, and data blocks with access frequencies in the bottom 30% to secondary partitions, and the initial state of the partitions is marked and recorded to generate an initial data partition table; S203: Based on the initial data partition table, the dynamic access situation of the data blocks during operation is captured by a real-time access monitoring method, the partition state is adjusted according to the latest access frequency, the data partition distribution is updated, and a dynamically adjusted partition table is generated; S204: Based on the dynamically adjusted partition table, the adjusted data partition results are written into the distributed storage nodes through the distributed node storage management method, the partition optimization storage is completed, and the optimized dynamic data partition table is generated.

[0008] Preferably, based on the optimized dynamic data partition table, an event-driven filtering algorithm is used to construct a Bloom filter to filter the key event data set that meets the conditions, and the data tags are judged item by item according to the defined rule table. The key event identifiers are marked for the data that hit the conditions and then extracted and stored; the specific steps of generating the key event data stream are: S301: Based on the optimized dynamic data partition table, a Bloom filter is constructed, a rule table of key events is defined, rule conditions are converted into a hash function of the Bloom filter, and potential event features of the data block are calculated using the formula: ; Mark data blocks as potential events by calculating item by item; Calculate the probability of potential events , generating potential event data after Bloom filtering; among them, Representative The potential event probability of each data block, Representative The weighted sum of the key eigenvalues ​​in the data blocks, Represents the mean of the weighted sum of the key feature values ​​of all data blocks, Represents the total number of data blocks, Represents the sum of squares of the weighted sum of the eigenvalues ​​of all data blocks and the mean value, Representative The regular hash value of a data block; S302: Based on the potential event data after Bloom filtering, the event feature values ​​in the data blocks are retrieved one by one through the key event matching method to determine whether the definition conditions of the rule table are met, and the data blocks that meet the conditions are screened to generate key event data blocks; S303: Based on the key event data block, an event priority labeling method is used to classify and label the key event data block, generate event priority labels according to event importance, and generate key event data with labeled priorities; S304: Based on the key event data with marked priorities, the data is classified and stored in the designated storage node according to the priorities through the data extraction and storage algorithm, and the extraction and storage of the key event data are completed to generate the key event data.

[0009] Preferably, based on the key event data stream, a heterogeneous computing task scheduling algorithm is adopted, by classifying and marking the computing density of task feature vectors, light tasks are assigned to CPU parallel queue processing according to the task load analysis results, and high-density tasks are assigned to GPU batch queue processing, and the distribution of computing tasks of each device is adjusted in combination with the dynamic load model; the specific steps of generating a set of computing results for efficient processing are: S401: Classifying task feature vectors based on key event data streams, marking tasks as lightweight tasks or intensive computing tasks by calculating the computational complexity and data dependency of the tasks, and generating task feature vectors marked by classification; S402: Based on the task feature vectors marked by classification, the current load and available computing power of each computing device are calculated through the load balancing analysis model, and the tasks are assigned to appropriate computing devices in combination with the task features to generate a task device allocation table; S403: Based on the task device allocation table, the lightweight tasks are allocated to the CPU parallel queue and the intensive computing tasks are allocated to the GPU batch computing queue through the parallel task scheduling algorithm, the task scheduling is completed, and the allocated computing task queue is generated; S404: Based on the assigned computing task queue, through the device execution and feedback mechanism, the processing results are output after the calculations are completed item by item, and a final task processing result set is generated to generate an efficient processing computing result set.

[0010] A data management system for reducing computing power usage, comprising the following modules: a feature extraction module, a dynamic partitioning module, an event filtering module, and a task scheduling module; The feature extraction module uses the L1 regularization constraint algorithm to calculate the feature sparsity column by column based on the high-dimensional data set, removes the feature items whose weights tend to zero, calculates the correlation between features in combination with the covariance matrix to extract the feature vector, and uses the matrix projection method to convert the high-dimensional data into the principal component space and remove the feature dimensions whose contribution rate is less than the preset value. After optimizing the matrix sparsity, the main features are output to generate a sparse low-dimensional data matrix; The feature extraction module includes a sparse processing submodule, a feature vector extraction submodule, and a principal component conversion submodule; The dynamic partitioning module is based on the sparse low-dimensional data matrix. It builds a data access frequency model by analyzing the access logs, uses the K-means clustering algorithm to allocate high-frequency access data to priority partitions and generate a partition table. It combines real-time monitoring to capture dynamic data access and adjust the partition status. It writes the adjusted partitions to the storage nodes through the distributed node management method to generate an optimized dynamic data partition table. The dynamic partitioning module includes a frequency modeling submodule, a partition clustering submodule, and a partition dynamic adjustment submodule; The event filtering module builds a Bloom filter based on the optimized dynamic data partition table to filter potential events. It marks potential events by calculating the data feature values ​​and matching them with the hash function of the event rules one by one. It uses the key event matching method to filter the data blocks that meet the conditions and mark the key events with priority tags. Finally, it uses the extraction and storage algorithm to store the event data to the target node according to the priority, generating a key event data stream. The event filtering module includes a Bloom filtering submodule, an event feature matching submodule, and a priority marking submodule; Task scheduling module: Based on the key event data stream, the task density and data dependency are calculated in combination with the task feature vector to classify the tasks. The available computing power of the computing device is calculated through the load balancing analysis model to dynamically generate a task allocation table. The parallel task scheduling algorithm is used to allocate lightweight tasks to the CPU parallel queue and high-density tasks to the GPU batch queue. After completing the task, the final result is output to generate a set of calculation results for efficient processing. The task scheduling module includes a task feature classification submodule, a task load balancing submodule, and a parallel queue scheduling submodule.

[0011] Preferably, the sparse processing submodule, based on a high-dimensional data set, adopts an L1 regularization constraint algorithm, constructs an objective function, takes the sparsity of the eigenvalue as a constraint condition, calculates the feature weights in the data matrix column by column, removes feature items whose weights tend to zero, and generates a preliminary sparse feature matrix; The feature vector extraction submodule uses the covariance matrix calculation method based on the preliminary sparse feature matrix to calculate the covariance values ​​between features to construct the feature correlation matrix, extracts the feature vector matrix through matrix decomposition, and selects the feature vectors with the highest weight ranking to generate the feature vector matrix; The principal component conversion submodule, based on the eigenvector matrix, projects the high-dimensional data into the principal component space through the matrix projection method, selects the feature dimensions according to the principal component contribution rate, removes the feature dimensions whose contribution rate is less than the preset threshold, optimizes the matrix sparsity and then outputs the result to generate a sparse low-dimensional data matrix.

[0012] Preferably, the frequency modeling submodule analyzes the data access log based on the sparse low-dimensional data matrix, calculates the access frequency and generates an access frequency ranking model by counting the number of accesses and time intervals of each data block, and generates a data access frequency model; The partition clustering submodule uses the K-means clustering algorithm based on the data access frequency model to cluster the access frequencies of data blocks, allocate the data blocks with the top 30% access frequencies to the priority partitions, and allocate the data blocks with the bottom 30% access frequencies to the secondary partitions, and generate the initial partition table; The partition dynamic adjustment submodule, based on the initial data partition table, captures the dynamic access status of data blocks through real-time monitoring, updates the access frequency data, adjusts the partition status and optimizes the partition structure according to the latest access frequency, and generates an optimized dynamic data partition table.

[0013] Preferably, the Bloom filter submodule constructs a Bloom filter based on the optimized dynamic data partition table, calculates the characteristic value hash function of each data block by defining a rule table of key events, marks the potential event data blocks that meet the rules as potential events, and generates potential event data after Bloom filtering; The event feature matching submodule uses the key event matching method based on the potential event data after Bloom filtering to retrieve the event feature values ​​one by one and determine whether they meet the conditions of the rule table, filter out the event data blocks that meet the conditions, and generate key event data blocks; The priority labeling submodule classifies the event priorities of the data blocks based on the key event data blocks, labels the events according to their importance, and generates key event data with priority labels.

[0014] Preferably, the task feature classification submodule classifies the task feature vectors by computational intensity based on the key event data with marked priorities, marks the tasks as lightweight tasks or intensive tasks according to their computational complexity and data dependency, and generates task feature vectors with classification marks; The task load balancing submodule calculates the current load and available computing power of each device based on the task feature vectors marked by classification and combined with the load balancing analysis model, generates a task and device allocation table, and generates a task and device allocation table; The parallel queue scheduling submodule uses a parallel task scheduling algorithm based on the task device allocation table to assign lightweight tasks to the CPU parallel queue and intensive tasks to the GPU batch queue to complete task scheduling and generate an assigned computing task queue.

[0015] The present invention provides a data management method and system for reducing computing power usage. It has the following beneficial effects: 1. The present invention innovatively uses L1 regularization to perform feature screening on high-dimensional data through the combination of sparse representation and principal component analysis, removes redundant data items and extracts principal component features with high correlation, effectively reduces data dimensions, reduces the consumption of computing resources, and retains the main information characteristics of the data, improving the efficiency and accuracy of subsequent processing. In dynamic partition optimization, by constructing a data access frequency model, combining the K-means clustering algorithm and the real-time dynamic adjustment mechanism, the locality of data access and the rationality of storage allocation are significantly improved, and the computational overhead and communication delay caused by cross-partition access are reduced. In the event filtering stage, the Bloom filter and the rule table hash function matching mechanism are introduced to achieve rapid screening of potential events, avoid repeated calculations of irrelevant data, further narrow the scope of data processing, and improve the accuracy and efficiency of data filtering. In task scheduling, through the computational intensity analysis of task features and the dynamic regulation of the load balancing model, the distributed scheduling of tasks and the efficient use of equipment computing power are realized, light tasks and intensive tasks are reasonably allocated, and resource conflicts and computing power waste are reduced. The overall processing logic forms a multi-level resource-saving mechanism from data input to task output through innovative methods of data simplification, partition optimization, event screening and task scheduling, achieving a significant effect of reducing computing power usage and providing a more efficient solution for high-performance computing and real-time computing scenarios.

[0016] 2. The present invention eliminates redundant features and extracts high-weight principal component features by combining the sparse processing of high-dimensional data sets with principal component analysis, thereby reducing data dimensions, improving computing resource utilization, and retaining core information, making subsequent data processing more efficient and accurate. Combining the data access frequency model with dynamic partition adjustment, the storage distribution and access path of the data are optimized through real-time monitoring and clustering of access features, significantly reducing the communication overhead and storage resource waste of cross-node access. In event data screening, the Bloom filter is used to perform fast hash calculation and rule matching on the data block feature values ​​to screen out potential key events, avoid resource occupation caused by full traversal, and further refine the data extraction range through event priority classification. In task scheduling, through the classification of task computing intensity and dynamic analysis of load balancing, light tasks and intensive tasks are respectively assigned to different device queues, significantly improving the allocation efficiency of computing resources and equipment utilization, and avoiding resource conflicts and task delays. The overall processing logic builds a highly targeted and efficient optimization mechanism in multiple links of the data processing chain, enabling the system to achieve significant computing power savings and performance improvements in large-scale data processing and real-time computing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the main steps of the present invention; Figure 2It is a schematic diagram of the refinement of S1 of the present invention; Figure 3 It is a schematic diagram of the refinement of S2 of the present invention; Figure 4 It is a schematic diagram of the refinement of S3 of the present invention; Figure 5 It is a schematic diagram of the refinement of S4 of the present invention; Figure 6 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0019] Please refer to the attached Figure 1 - Attachment Figure 6 , an embodiment of the present invention provides a data management method for reducing computing power usage, comprising the following steps: S1: Based on data distribution analysis, the sparse representation algorithm and principal component analysis method are used to eliminate the redundancy of high-dimensional data features by setting L1 regularization constraints, decompose the data correlation matrix by combining matrix operations, extract the feature vectors with the highest contribution value and convert them into sparse low-dimensional matrices; generate a sparse low-dimensional data matrix; Based on data distribution analysis and combined with matrix construction of high-dimensional data features, the feature sparsity of the data matrix is ​​calculated column by column through the L1 regularization constraint algorithm, and the feature items with weight values ​​close to zero are eliminated. At the same time, a covariance matrix is ​​constructed to quantify the correlation between features. Matrix decomposition is combined to extract eigenvectors and their ranking of contribution to the overall data is calculated. The data is converted into principal component space through matrix projection and eigenvectors with larger contribution rates are screened. The sparse matrix is ​​optimized to retain the main features and eliminate low-weight features. Finally, a sparse low-dimensional data matrix is ​​generated through the optimized matrix.

[0020] S2: Based on the sparse low-dimensional data matrix, a dynamic partition optimization algorithm is used to select high-frequency access data sets by building a data access frequency distribution model. The data is allocated to different storage partitions according to weights in combination with the distributed node task table. The partition weights are dynamically adjusted based on real-time monitoring and the allocation results are updated; an optimized dynamic data partition table is generated; Based on the sparse low-dimensional data matrix, a data access frequency model is constructed by analyzing data access log records. The number of accesses and time intervals are counted in units of data blocks, and the access frequency distribution is calculated. The access frequency is partitioned using the K-means clustering method. High-access frequency data is assigned to priority partitions and an initial partition table is generated. At the same time, data access is monitored in real time to dynamically adjust the access frequency data. The adjusted partition status is stored in distributed nodes. The cross-partition access communication cost is reduced by optimizing the partition structure and written into the optimized dynamic data partition table.

[0021] S3: Based on the optimized dynamic data partition table, an event-driven filtering algorithm is used to construct a Bloom filter to filter the key event data set that meets the conditions. The data tags are judged item by item according to the defined rule table. The key event identifiers are marked for the data that hits the conditions and then extracted and stored; the key event data stream is generated; In the above content, by constructing a Bloom filter, the data block feature values ​​are hashed and potential events are screened, and the probability of potential events is calculated according to the formula .

[0022] In the formula, Representative The potential event probability of each data block, Representative The weighted sum of the eigenvalues ​​of data blocks, represents the mean of the weighted sum of the eigenvalues ​​of all data blocks, is the total number of data blocks, represents the sum of squared variances of the weighted sum of eigenvalues ​​and the mean, For the The regular hash value of a data block.

[0023] S4: Based on the key event data stream, a heterogeneous computing task scheduling algorithm is adopted. By classifying and marking the computing intensity of task feature vectors, light tasks are assigned to the CPU parallel queue for processing and high-intensity tasks are assigned to the GPU batch queue for processing according to the task load analysis results. The distribution of computing tasks of each device is adjusted in combination with the dynamic load model; a set of computing results for efficient processing is generated.

[0024] Based on the key event data stream, the task feature vector is extracted and the density and data dependency of each task are calculated. Lightweight tasks and intensive tasks are classified and marked. Combined with the current computing power of the device and the task load status, the task and device allocation table is dynamically generated through the load balancing model. At the same time, the parallel scheduling algorithm is called to add lightweight tasks to the CPU task queue and intensive tasks to the GPU task queue for batch processing. Tasks are completed one by one according to the device execution feedback and the final calculation results are generated.

[0025] Based on data distribution analysis, the sparse representation algorithm and principal component analysis method are used to eliminate the redundancy of high-dimensional data features by setting L1 regularization constraints, decompose the data correlation matrix by combining matrix operations, extract the feature vectors with the highest contribution value and convert them into sparse low-dimensional matrices; the specific steps for generating a sparse low-dimensional data matrix are as follows: S101: Based on the high-dimensional data set, the L1 regularization constraint algorithm is used to construct the objective function, take the sparsity of the eigenvalue as the constraint condition, calculate each feature in the data matrix column by column, remove the feature items whose weight value tends to zero under the constraint condition, and generate a preliminary sparse feature matrix; Based on the sparse low-dimensional data matrix, by analyzing the data access log records, the number of accesses and access time intervals of each data block are counted, and the data access frequency is calculated using the time series analysis method. ; in, For data blocks The frequency of visits, is the number of accesses to the data block, For the time interval, a data access frequency model is generated by sorting the access frequencies to generate a data access frequency model.

[0026] S102: Based on the preliminary sparse feature matrix, a covariance matrix calculation method is used to construct a feature correlation matrix according to the covariance values ​​between features, a feature vector matrix is ​​obtained through feature decomposition, and the feature vectors with the highest weight ranking are extracted as principal components to generate a feature vector matrix; S103: Based on the eigenvector matrix, the high-dimensional data is projected into the principal component space by a matrix projection method, the feature dimension is reduced according to the principal component contribution rate, and the feature dimension whose principal component contribution rate is less than a preset threshold is eliminated to generate a principal component low-dimensional matrix; S104: Based on the principal component low-dimensional matrix, a sparse matrix mapping method is used to optimize the matrix sparsity, retain the sparse features of the principal component weights in the top 30%, output the final sparse matrix result, and generate a sparse low-dimensional data matrix.

[0027] In S101, the sparsity is optimized by eliminating the feature items whose weight values ​​tend to zero, and the generated preliminary sparse feature matrix provides a basis for subsequent feature correlation calculation; In S102, the preliminary sparse feature matrix is ​​called, the relationship between features is analyzed through the covariance matrix, and the feature vector matrix is ​​constructed to provide support for high-dimensional data projection; In S103, based on the eigenvector matrix, invalid features are reduced by screening the principal component contribution rate, and the generated principal component low-dimensional matrix provides a data basis for sparse optimization; In S104, the principal component low-dimensional matrix is ​​called, and the final sparse low-dimensional data matrix is ​​generated through sparse optimization to form a complete feature optimization chain.

[0028] Based on the sparse low-dimensional data matrix, a dynamic partition optimization algorithm is used to select high-frequency access data sets by building a data access frequency distribution model. Combined with the distributed node task table, the data is allocated to different storage partitions according to weights. The partition weights are dynamically adjusted based on real-time monitoring and the allocation results are updated. The specific steps to generate the optimized dynamic data partition table are as follows: S201: constructing a data access frequency model based on a sparse low-dimensional data matrix, counting the number of accesses and time intervals of each data block according to data access log records, calculating the access frequency and generating a data block access frequency ranking, and generating a data access frequency model; S202: Based on the data access frequency model, a K-means clustering algorithm is used to allocate data blocks with access frequencies in the top 30% to priority partitions, and data blocks with access frequencies in the bottom 30% to secondary partitions, and the initial state of the partitions is marked and recorded to generate an initial data partition table; S203: Based on the initial data partition table, the dynamic access situation of the data blocks during operation is captured by a real-time access monitoring method, the partition state is adjusted according to the latest access frequency, the data partition distribution is updated, and a dynamically adjusted partition table is generated; S204: Based on the dynamically adjusted partition table, the adjusted data partition results are written into the distributed storage nodes through the distributed node storage management method, the partition optimization storage is completed, and the optimized dynamic data partition table is generated.

[0029] In S201, an access frequency model is calculated based on the sparse low-dimensional data matrix to provide reference data for subsequent partition division; In S202, the data access frequency model is called to generate an initial data partition table through K-means clustering to provide an initial partition basis for dynamic adjustment; In S203, the initial data partition table is called and the partition status is updated in combination with the real-time monitoring data to generate a dynamically adjusted partition table; In S204, the dynamically adjusted partition table is called to write the partition result into the distributed node storage, forming a complete partition optimization process, and generating a dynamic data partition table after the final result optimization.

[0030] Based on the optimized dynamic data partition table, an event-driven filtering algorithm is used to construct a Bloom filter to filter the key event data set that meets the conditions. The data tags are judged item by item according to the defined rule table. The key event identifiers are marked for the data that hits the conditions and then extracted and stored. The specific steps for generating the key event data stream are as follows: S301: Based on the optimized dynamic data partition table, a Bloom filter is constructed, a rule table of key events is defined, rule conditions are converted into a hash function of the Bloom filter, and potential event features of the data block are calculated using the formula: ; Mark data blocks as potential events by calculating item by item; Calculate the probability of potential events , generate potential event data after Bloom filtering; in, Representative The potential event probability of each data block, Representative The weighted sum of the key eigenvalues ​​in the data blocks, Represents the mean of the weighted sum of the key feature values ​​of all data blocks, Represents the total number of data blocks, Represents the sum of squares of the weighted sum of the eigenvalues ​​of all data blocks and the mean value, Representative The regular hash value of a data block; S302: Based on the potential event data after Bloom filtering, the event feature values ​​in the data blocks are retrieved one by one through the key event matching method to determine whether the definition conditions of the rule table are met, and the data blocks that meet the conditions are screened to generate key event data blocks; S303: Based on the key event data block, an event priority labeling method is used to classify and label the key event data block, generate event priority labels according to event importance, and generate key event data with labeled priorities; S304: Based on the key event data with marked priorities, the data is classified and stored in the designated storage node according to the priorities through the data extraction and storage algorithm, and the extraction and storage of the key event data are completed to generate the key event data.

[0031] In S301, a Bloom filter is constructed based on the optimized dynamic data partition table and the potential event probability is calculated. The generated Bloom-filtered potential event data provides an initial screening set for subsequent screening; In S302, the potential event data after Bloom filtering is called, and the matching and screening are performed item by item in combination with the conditions of the rule table, and the generated key event data blocks provide a screening basis for subsequent classification and labeling; In S303, based on the key event data block, priority-labeled key event data is generated through priority classification labeling to provide a classification basis for subsequent storage optimization; In S304, the key event data with marked priorities is called, and the data is classified and stored in the designated nodes through the storage algorithm, and finally a key event data stream is formed, completing the whole process of event data processing from screening to storage.

[0032] Based on the key event data stream, a heterogeneous computing task scheduling algorithm is adopted. By classifying and marking the computing density of task feature vectors, light tasks are assigned to CPU parallel queue processing according to the task load analysis results, and high-density tasks are assigned to GPU batch queue processing. The distribution of computing tasks on each device is adjusted in combination with the dynamic load model. The specific steps to generate a set of computing results for efficient processing are as follows: S401: Classifying task feature vectors based on key event data streams, marking tasks as lightweight tasks or intensive computing tasks by calculating the computational complexity and data dependency of the tasks, and generating task feature vectors marked by classification; S402: Based on the task feature vectors marked by classification, the current load and available computing power of each computing device are calculated through the load balancing analysis model, and the tasks are assigned to appropriate computing devices in combination with the task features to generate a task device allocation table; S403: Based on the task device allocation table, the lightweight tasks are allocated to the CPU parallel queue and the intensive computing tasks are allocated to the GPU batch computing queue through the parallel task scheduling algorithm, the task scheduling is completed, and the allocated computing task queue is generated; S404: Based on the assigned computing task queue, through the device execution and feedback mechanism, the processing results are output after the calculations are completed item by item, and a final task processing result set is generated to generate an efficient processing computing result set.

[0033] In S401, the task feature vector is parsed based on the key event data stream and the density is calculated, and the generated classification-labeled task feature vector provides a classification basis for task allocation; In S402, the task feature vector of the classification mark is called and combined with the load balancing analysis to generate a task device allocation table to provide a device matching basis for task scheduling; In S403, tasks are assigned to different device queues using a parallel task scheduling algorithm based on the task device allocation table, and the generated assigned computing task queue provides a clear queue structure for task execution; In S404, the allocated computing task queue is called to complete the task execution and integrate all computing results, and finally generate a computing result set with efficient processing.

[0034] A data management system for reducing computing power usage, characterized by comprising the following modules: a feature extraction module, a dynamic partitioning module, an event filtering module, and a task scheduling module; The feature extraction module uses the L1 regularization constraint algorithm to calculate the feature sparsity column by column based on the high-dimensional data set, removes the feature items whose weights tend to zero, calculates the correlation between features in combination with the covariance matrix to extract the feature vector, and uses the matrix projection method to convert the high-dimensional data into the principal component space and remove the feature dimensions whose contribution rate is less than the preset value. After optimizing the matrix sparsity, the main features are output to generate a sparse low-dimensional data matrix; In the feature extraction module, the sparsification processing submodule performs weight elimination to generate a preliminary sparse feature matrix, the eigenvector extraction submodule calculates feature correlation and generates a eigenvector matrix, and the principal component conversion submodule uses the matrix projection method to screen features with higher contribution rates to generate a sparse low-dimensional data matrix.

[0035] The feature extraction module includes a sparse processing submodule, a feature vector extraction submodule, and a principal component conversion submodule; The dynamic partitioning module is based on the sparse low-dimensional data matrix. It builds a data access frequency model by analyzing the access logs, uses the K-means clustering algorithm to allocate high-frequency access data to priority partitions and generate a partition table. It combines real-time monitoring to capture dynamic data access and adjust the partition status. It writes the adjusted partitions to the storage nodes through the distributed node management method to generate an optimized dynamic data partition table. In the dynamic partitioning module, the frequency modeling submodule analyzes the access log to generate a data access frequency model, the partition clustering submodule performs clustering calculations based on the access frequency and generates an initial data partition table, and the partition dynamic adjustment submodule monitors access in real time, dynamically optimizes the partition structure, and generates an optimized dynamic data partition table.

[0036] The dynamic partitioning module includes a frequency modeling submodule, a partition clustering submodule, and a partition dynamic adjustment submodule; The event filtering module builds a Bloom filter based on the optimized dynamic data partition table to filter potential events. It marks potential events by calculating the data feature values ​​and matching them with the hash function of the event rules one by one. It uses the key event matching method to filter the data blocks that meet the conditions and mark the key events with priority tags. Finally, it uses the extraction and storage algorithm to store the event data to the target node according to the priority, generating a key event data stream. In the event filtering module, the Bloom filtering submodule filters potential event data blocks to generate potential event data after Bloom filtering, the event feature matching submodule filters data blocks that meet the conditions of the rule table to generate key event data blocks, and the priority labeling submodule classifies and labels the event data priorities to generate key event data labeled with priorities.

[0037] The event filtering module includes a Bloom filtering submodule, an event feature matching submodule, and a priority marking submodule; Task scheduling module: Based on the key event data stream, the task density and data dependency are calculated in combination with the task feature vector to classify the tasks. The available computing power of the computing device is calculated through the load balancing analysis model to dynamically generate a task allocation table. The parallel task scheduling algorithm is used to allocate lightweight tasks to the CPU parallel queue and high-density tasks to the GPU batch queue. After completing the task, the final result is output to generate a set of calculation results for efficient processing. In the task scheduling module, the task feature classification submodule generates a classified marked task feature vector by calculating the task feature vector classification, the task load balancing submodule generates a task device allocation table based on the device load and computing power, and the parallel queue scheduling submodule generates an allocated computing task queue through task scheduling.

[0038] The task scheduling module includes a task feature classification submodule, a task load balancing submodule, and a parallel queue scheduling submodule.

[0039] The sparse processing submodule is based on high-dimensional data sets and adopts the L1 regularization constraint algorithm. By constructing the objective function, the sparsity of the eigenvalue is used as a constraint condition, the feature weights in the data matrix are calculated column by column, and the feature items with weights approaching zero are eliminated to generate a preliminary sparse feature matrix. Based on high-dimensional data sets, the L1 regularization constraint algorithm is used to construct the objective function ; in, is the feature weight vector, is the data matrix, is the target vector, The objective function is optimized iteratively by using the gradient descent method to regularize the parameter. The feature weights in the data matrix are calculated and screened column by column. The features whose weight values ​​tend to zero are removed, and the features with significant importance are retained, thereby optimizing the feature sparsity.

[0040] The feature vector extraction submodule uses the covariance matrix calculation method based on the preliminary sparse feature matrix to calculate the covariance values ​​between features to construct the feature correlation matrix, extracts the feature vector matrix through matrix decomposition, and selects the feature vectors with the highest weight ranking to generate the feature vector matrix; Based on the preliminary sparse feature matrix, the covariance matrix calculation method is adopted, and the formula is used ; in, is the covariance matrix, For each column of data in the matrix, The feature correlation matrix is ​​constructed according to the covariance value between features, and the eigendecomposition algorithm is used to decompose it, extract the eigenvector matrix, and select the eigenvectors with the highest weight by sorting the eigenvalues, and retain the principal component eigenvectors for the next step of optimization.

[0041] The principal component conversion submodule, based on the eigenvector matrix, projects the high-dimensional data into the principal component space through the matrix projection method, selects the feature dimensions according to the principal component contribution rate, removes the feature dimensions whose contribution rate is less than the preset threshold, optimizes the matrix sparsity and then outputs the result to generate a sparse low-dimensional data matrix.

[0042] Based on the eigenvector matrix, the high-dimensional data is mapped to the principal component space through the matrix projection method, using the formula ; in, is the projection result, is the initial matrix, is the eigenvector matrix, and the principal component contribution rate is calculated according to the formula Eliminate the dimensions whose principal component contribution rate is lower than the preset threshold, reduce redundant information, further improve the low-dimensional representation ability of data through sparse optimization, and form the final sparse low-dimensional matrix after optimizing the matrix sparsity; The preliminary sparse feature matrix generated by the sparse processing submodule provides basic data for subsequent feature correlation calculations, significantly reducing the interference of useless features; The eigenvector extraction submodule uses the preliminary sparse feature matrix to construct the covariance matrix and extract the eigenvectors. The generated eigenvector matrix is ​​used as the basis for principal component transformation. The principal component conversion submodule calls the eigenvector matrix for high-dimensional projection and sparse optimization, and finally outputs a sparse low-dimensional data matrix to form a complete feature processing chain.

[0043] The frequency modeling submodule analyzes data access logs based on the sparse low-dimensional data matrix, calculates the access frequency and generates an access frequency ranking model by counting the number of accesses and time intervals of each data block, and generates a data access frequency model; The partition clustering submodule uses the K-means clustering algorithm based on the data access frequency model to cluster the access frequencies of data blocks, allocate the data blocks with the top 30% access frequencies to the priority partitions, and allocate the data blocks with the bottom 30% access frequencies to the secondary partitions, and generate the initial partition table; The partition dynamic adjustment submodule, based on the initial data partition table, captures the dynamic access status of data blocks through real-time monitoring, updates the access frequency data, adjusts the partition status and optimizes the partition structure according to the latest access frequency, and generates an optimized dynamic data partition table.

[0044] The frequency modeling submodule generates a data access frequency model by analyzing the access logs, providing quantitative indicators for the subsequent partition priority division; The partition clustering submodule calls the data access frequency model, and the initial data partition table generated by the K-means clustering algorithm provides the initial partition structure for dynamic adjustment; The partition dynamic adjustment submodule is based on the initial data partition table, combines real-time monitoring to update the access frequency and optimize the partition structure, and finally generates an optimized dynamic data partition table, realizing the optimization of dynamic partitions and the rational allocation of resources.

[0045] The Bloom filter submodule builds a Bloom filter based on the optimized dynamic data partition table. By defining the rule table of key events, it calculates the characteristic value hash function of each data block, marks the potential event data blocks that meet the rules as potential events, and generates potential event data after Bloom filtering; The event feature matching submodule uses the key event matching method based on the potential event data after Bloom filtering to retrieve the event feature values ​​one by one and determine whether they meet the conditions of the rule table, filter out the event data blocks that meet the conditions, and generate key event data blocks; The priority labeling submodule classifies the event priorities of the data blocks based on the key event data blocks, labels the events according to their importance, and generates key event data with priority labels.

[0046] The Bloom filter submodule performs hash calculation and rule matching on the characteristic values ​​of data blocks in the optimized dynamic data partition table. The generated Bloom filtered potential event data provides a basis for subsequent screening. The event feature matching submodule calls the potential event data after Bloom filtering, and selects the key event data blocks that meet the conditions by comparing the feature values ​​one by one; The priority labeling submodule is based on the key event data block. It classifies the data by calculating the importance of the event, generates key event data with labeled priorities, and realizes hierarchical management and optimization of key events.

[0047] The task feature classification submodule classifies the computational density of task feature vectors based on the key event data with marked priorities, marks them as lightweight tasks or intensive tasks according to their computational complexity and data dependencies, and generates task feature vectors with classification labels. Based on the key event data with marked priorities, the feature vector of the task is extracted and the computational intensity analysis method is used. According to the formula ; in, For the The intensity of the task, is the computational complexity of the task, indicating the total amount of computing resources required to complete the task is the data dependency of the task, indicating the number of accesses to other data blocks during the execution of the task. Tasks are classified according to the density results, and tasks with higher density are marked as dense tasks, and tasks with lower density are marked as lightweight tasks.

[0048] The task load balancing submodule calculates the current load and available computing power of each device based on the task feature vectors marked by classification and combined with the load balancing analysis model, generates a task and device allocation table, and generates a task and device allocation table; Based on the task feature vector of classification labeling and combined with the load balancing analysis model, the current load and available computing power of each device are calculated. The load calculation formula is as follows: ; in, For equipment The load utilization rate, Indicates the device The total weight of the assigned tasks, is the total computing power of the device. Tasks are assigned to low-load devices first, and intensive tasks are assigned to high-computing devices first, and a task and device allocation table is generated.

[0049] The parallel queue scheduling submodule uses a parallel task scheduling algorithm based on the task device allocation table to assign lightweight tasks to the CPU parallel queue and intensive tasks to the GPU batch queue to complete task scheduling and generate an assigned computing task queue.

[0050] Based on the task device allocation table, a parallel task scheduling algorithm is adopted to build a task queue model, which assigns lightweight tasks to the CPU's parallel task queue. The queue priority model is used to dynamically adjust the execution order of lightweight tasks, and a batch task processing mechanism is built to assign intensive tasks to the GPU's batch computing queue. The scheduling strategy that maximizes the use of device computing power optimizes resource allocation and completes task scheduling.

[0051] The task feature classification submodule generates a classification-labeled task feature vector through computational density analysis, providing a clear classification basis for task allocation and scheduling; The task load balancing submodule calls the task feature vector marked by classification, combines the load and computing power of the equipment to allocate resources, and generates a task device allocation table to provide an allocation plan for the actual scheduling of the task; The parallel queue scheduling submodule is based on the task device allocation table. Through task priority scheduling and queue management, it finally generates the allocated computing task queue and completes the scheduling and allocation optimization of tasks.

[0052] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection, an electrical connection, or communication with each other; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0053] Obviously, the embodiments described above are only some embodiments of the present invention, rather than all embodiments. The preferred embodiments of the present invention are given in the accompanying drawings, but they do not limit the patent scope of the present invention. The present invention can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive. Although the present invention has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific embodiments, or to perform equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of the present invention, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present invention.

Claims

1. A data management method for reducing computing power usage, characterized in that: The following steps are involved: S1: Based on data distribution analysis, the sparse representation algorithm and principal component analysis method are used to eliminate the redundancy of high-dimensional data features by setting L1 regularization constraints, decompose the data correlation matrix by combining matrix operations, extract the feature vectors with the top 30% contribution rate and convert them into sparse low-dimensional matrices; Generate a sparse low-dimensional data matrix; S2: Based on the sparse low-dimensional data matrix, a dynamic partition optimization algorithm is used to select high-frequency access data sets with more than 10 accesses per minute by building a data access frequency distribution model. The data is then allocated to different storage partitions according to weights in combination with the distributed node task table. The partition weights are dynamically adjusted based on real-time monitoring and the allocation results are updated. Generate optimized dynamic data partition table; S3: Based on the optimized dynamic data partition table, an event-driven filtering algorithm is used to construct a Bloom filter to filter key event data sets that meet the conditions. Data tags are judged item by item according to the defined rule table, and key event identifiers are marked for data that hits the conditions before extraction and storage. Generate key event data streams; S4: Based on the key event data stream, a heterogeneous computing task scheduling algorithm is adopted. By classifying and marking the computing intensity of task feature vectors, light tasks with computing complexity lower than a specific threshold are assigned to the CPU parallel queue for processing according to the task load analysis results, and high-intensity tasks with computing complexity higher than a specific threshold are assigned to the GPU batch queue for processing. The distribution of computing tasks of each device is adjusted in combination with the dynamic load model; a set of computing results for efficient processing is generated.

2. A data management method for reducing computing power usage according to claim 1, characterized in that: Based on data distribution analysis, the sparse representation algorithm and principal component analysis method are used to eliminate the redundancy of high-dimensional data features by setting L1 regularization constraints, and the data correlation matrix is ​​decomposed by combining matrix operations to extract the feature vectors with the top 30% contribution rate and convert them into sparse low-dimensional matrices. The specific steps to generate a sparse low-dimensional data matrix are: S101: Based on the high-dimensional data set, the L1 regularization constraint algorithm is used to construct the objective function, take the sparsity of the eigenvalue as the constraint condition, calculate each feature in the data matrix column by column, remove the feature items with weight values ​​less than 0.01 under the constraint condition, and generate a preliminary sparse feature matrix; S102: Based on the preliminary sparse feature matrix, a covariance matrix calculation method is used to construct a feature correlation matrix according to the covariance values ​​between features, and a feature vector matrix is ​​obtained through feature decomposition. The feature vectors ranked in the top 30% of the weights are extracted as principal components to generate a feature vector matrix. S103: Based on the eigenvector matrix, the high-dimensional data is projected into the principal component space by a matrix projection method, the feature dimension is reduced according to the principal component contribution rate, the feature dimension with a principal component contribution rate less than 10% is eliminated, and a principal component low-dimensional matrix is ​​generated; S104: Based on the principal component low-dimensional matrix, a sparse matrix mapping method is used to optimize the matrix sparsity, retain the sparse features of the principal component weights in the top 30%, output the final sparse matrix result, and generate a sparse low-dimensional data matrix.

3. A data management method for reducing computing power usage according to claim 1, characterized in that: Based on the sparse low-dimensional data matrix, a dynamic partition optimization algorithm is used to construct a data access frequency distribution model to screen high-frequency access data sets with more than 10 accesses per minute. Combined with the distributed node task table, the data is allocated to different storage partitions according to weights. The partition weights are dynamically adjusted based on real-time monitoring and the allocation results are updated. The specific steps to generate the optimized dynamic data partition table are as follows: S201: constructing a data access frequency model based on a sparse low-dimensional data matrix, counting the number of accesses and time intervals of each data block according to data access log records, calculating the access frequency and generating a data block access frequency ranking, and generating a data access frequency model; S202: Based on the data access frequency model, a K-means clustering algorithm is used to allocate data blocks with access frequencies in the top 30% to priority partitions, and data blocks with access frequencies in the bottom 30% to secondary partitions, and the initial state of the partitions is marked and recorded to generate an initial data partition table; S203: Based on the initial data partition table, the dynamic access situation of the data blocks during operation is captured by a real-time access monitoring method, the partition state is adjusted according to the latest access frequency, the data partition distribution is updated, and a dynamically adjusted partition table is generated; S204: Based on the dynamically adjusted partition table, the adjusted data partition results are written into the distributed storage nodes through the distributed node storage management method, the partition optimization storage is completed, and the optimized dynamic data partition table is generated.

4. A data management method for reducing computing power usage according to claim 1, characterized in that: Based on the optimized dynamic data partition table, an event-driven filtering algorithm is used to construct a Bloom filter to filter the key event data set that meets the conditions. The data tags are judged item by item according to the defined rule table. The key event identifiers are marked for the data that hits the conditions and then extracted and stored. The specific steps for generating the key event data stream are as follows: S301: Based on the optimized dynamic data partition table, a Bloom filter is constructed, a rule table of key events is defined, rule conditions are converted into a hash function of the Bloom filter, and potential event features of the data block are calculated using the formula: ; Mark data blocks as potential events by calculating item by item; Calculate the probability of potential events , generating potential event data after Bloom filtering; among them, Representative The potential event probability of each data block, Representative The weighted sum of the key eigenvalues ​​in the data blocks, Represents the mean of the weighted sum of the key feature values ​​of all data blocks, Represents the total number of data blocks, Represents the sum of squares of the weighted sum of the eigenvalues ​​of all data blocks and the mean value, Representative The regular hash value of a data block; S302: Based on the potential event data after Bloom filtering, the event feature values ​​in the data blocks are retrieved one by one through the key event matching method to determine whether the definition conditions of the rule table are met, and the data blocks that meet the conditions are screened to generate key event data blocks; S303: Based on the key event data block, an event priority labeling method is used to classify and label the key event data block, generate event priority labels according to event importance, and generate key event data with labeled priorities; S304: Based on the key event data with marked priorities, the data is classified and stored in the designated storage node according to the priorities through the data extraction and storage algorithm, the extraction and storage of the key event data is completed, and the key event data stream is generated.

5. A data management method for reducing computing power usage according to claim 1, characterized in that: Based on the key event data stream, a heterogeneous computing task scheduling algorithm is adopted. By classifying and marking the computing density of task feature vectors, light tasks are assigned to CPU parallel queue processing according to the task load analysis results, and high-density tasks are assigned to GPU batch queue processing. The distribution of computing tasks on each device is adjusted in combination with the dynamic load model. The specific steps to generate a set of computing results for efficient processing are as follows: S401: Classifying task feature vectors based on key event data streams, marking tasks as lightweight tasks or intensive computing tasks by calculating the computational complexity and data dependency of the tasks, and generating task feature vectors marked by classification; S402: Based on the task feature vectors marked by classification, the current load and available computing power of each computing device are calculated through the load balancing analysis model, and the tasks are assigned to the corresponding computing devices in combination with the task features to generate a task device allocation table; S403: Based on the task device allocation table, the lightweight tasks are allocated to the CPU parallel queue and the intensive computing tasks are allocated to the GPU batch computing queue through the parallel task scheduling algorithm, the task scheduling is completed, and the allocated computing task queue is generated; S404: Based on the assigned computing task queue, through the device execution and feedback mechanism, the processing results are output after the calculations are completed item by item, and a final task processing result set is generated to generate an efficient processing computing result set.

6. A data management system for reducing computing power usage, characterized in that: Includes the following modules: feature extraction module, dynamic partitioning module, event filtering module, task scheduling module; The feature extraction module uses the L1 regularization constraint algorithm to calculate the feature sparsity column by column based on the high-dimensional data set, removes the feature items whose weights tend to zero, calculates the correlation between features in combination with the covariance matrix to extract the feature vector, and uses the matrix projection method to convert the high-dimensional data into the principal component space and remove the feature dimensions whose contribution rate is less than the preset value. After optimizing the matrix sparsity, the main features are output to generate a sparse low-dimensional data matrix; The feature extraction module includes a sparse processing submodule, a feature vector extraction submodule, and a principal component conversion submodule; The dynamic partitioning module is based on the sparse low-dimensional data matrix. It builds a data access frequency model by analyzing the access logs, uses the K-means clustering algorithm to allocate high-frequency access data to priority partitions and generate a partition table. It combines real-time monitoring to capture dynamic data access and adjust the partition status. It writes the adjusted partitions to the storage nodes through the distributed node management method to generate an optimized dynamic data partition table. The dynamic partitioning module includes a frequency modeling submodule, a partition clustering submodule, and a partition dynamic adjustment submodule; The event filtering module builds a Bloom filter based on the optimized dynamic data partition table to filter potential events. It marks potential events by calculating the data feature values ​​and matching them with the hash function of the event rules one by one. It uses the key event matching method to filter the data blocks that meet the conditions and mark the key events with priority tags. Finally, it uses the extraction and storage algorithm to store the event data to the target node according to the priority, generating a key event data stream. The event filtering module includes a Bloom filtering submodule, an event feature matching submodule, and a priority marking submodule; Task scheduling module; Based on the key event data stream, the task density and data dependency are calculated in combination with the task feature vector to classify the tasks. The available computing power of the computing device is calculated through the load balancing analysis model to dynamically generate a task allocation table. The parallel task scheduling algorithm is used to allocate lightweight tasks to the CPU parallel queue and high-density tasks to the GPU batch queue. After the task is completed, the final result is output to generate a set of calculation results for efficient processing. The task scheduling module includes a task feature classification submodule, a task load balancing submodule, and a parallel queue scheduling submodule.

7. A data management system for reducing computing power usage according to claim 6, characterized in that: The sparse processing submodule is based on high-dimensional data sets and adopts the L1 regularization constraint algorithm. By constructing the objective function, the sparsity of the eigenvalue is used as a constraint condition, the feature weights in the data matrix are calculated column by column, and the feature items with weights approaching zero are eliminated to generate a preliminary sparse feature matrix. The feature vector extraction submodule uses the covariance matrix calculation method based on the preliminary sparse feature matrix to calculate the covariance values ​​between features to construct the feature correlation matrix, extracts the feature vector matrix through matrix decomposition, and selects the feature vectors with the highest weight ranking to generate the feature vector matrix; The principal component conversion submodule, based on the eigenvector matrix, projects the high-dimensional data into the principal component space through the matrix projection method, selects the feature dimensions according to the principal component contribution rate, removes the feature dimensions whose contribution rate is less than the preset threshold, optimizes the matrix sparsity and then outputs the result to generate a sparse low-dimensional data matrix.

8. A data management system for reducing computing power usage according to claim 6, characterized in that: The frequency modeling submodule analyzes data access logs based on the sparse low-dimensional data matrix, calculates the access frequency and generates an access frequency ranking model by counting the number of accesses and time intervals of each data block, and generates a data access frequency model; The partition clustering submodule uses the K-means clustering algorithm based on the data access frequency model to cluster the access frequencies of data blocks, allocate the data blocks with the top 30% access frequencies to the priority partitions, and allocate the data blocks with the bottom 30% access frequencies to the secondary partitions, and generate the initial partition table; The partition dynamic adjustment submodule, based on the initial data partition table, captures the dynamic access status of data blocks through real-time monitoring, updates the access frequency data, adjusts the partition status and optimizes the partition structure according to the latest access frequency, and generates an optimized dynamic data partition table.

9. A data management system for reducing computing power usage according to claim 6, characterized in that: The Bloom filter submodule builds a Bloom filter based on the optimized dynamic data partition table. By defining the rule table of key events, it calculates the characteristic value hash function of each data block, marks the potential event data blocks that meet the rules as potential events, and generates potential event data after Bloom filtering; The event feature matching submodule uses the key event matching method based on the potential event data after Bloom filtering to retrieve the event feature values ​​one by one and determine whether they meet the conditions of the rule table, filter out the event data blocks that meet the conditions, and generate key event data blocks; The priority labeling submodule classifies the event priorities of the data blocks based on the key event data blocks, labels the events according to their importance, and generates key event data with priority labels.

10. A data management system for reducing computing power usage according to claim 6, characterized in that: The task feature classification submodule classifies the computational density of task feature vectors based on the key event data with marked priorities, marks them as lightweight tasks or intensive tasks according to their computational complexity and data dependencies, and generates task feature vectors with classification labels. The task load balancing submodule calculates the current load and available computing power of each device based on the task feature vectors marked by classification and combined with the load balancing analysis model, generates a task and device allocation table, and generates a task and device allocation table; The parallel queue scheduling submodule uses a parallel task scheduling algorithm based on the task device allocation table to assign lightweight tasks to the CPU parallel queue and intensive tasks to the GPU batch queue to complete task scheduling and generate an assigned computing task queue.

Images