A data mining-based method, system, and storage medium for millet gene screening.

By using data mining-based methods, intelligent clustering and dynamic load balancing of millet genetic data were achieved, solving the problems of unbalanced computing node load and low resource utilization, and improving analysis efficiency and result accuracy.

CN121963847BActive Publication Date: 2026-11-17ANYANG ACAD OF AGRI SCI +1
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202610084594.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-01-22
Publication Date
2026-11-17
Estimated Expiration
2046-01-22

AI Technical Summary

Technical Problem

Existing technologies in millet genomics research suffer from problems such as unbalanced computing node load, low resource utilization, and poor data consistency, which limit the efficiency of analysis and the accuracy of results.

Method used

By employing a data mining-based approach, intelligent clustering based on density features and a dynamic load assessment mechanism, combined with dynamic migration adjustment, we achieve precise matching of computing resources and real-time monitoring of task execution, and construct a multi-level quality assurance mechanism to ensure result consistency.

Benefits of technology

It significantly improved the utilization of computing resources, shortened the analysis cycle, maintained high throughput, and ensured the accuracy and reliability of millet gene screening results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121963847B_ABST
    Figure CN121963847B_ABST
Patent Text Reader

Abstract

This application relates to the fields of bioinformatics and distributed computing technology, and discloses a method, system, and storage medium for millet gene screening based on data mining. The method includes: acquiring millet gene data, performing initial clustering processing, and dividing it into multiple data segments to obtain a segment set; evaluating the resource requirement indicators of each data segment in the segment set; acquiring the current load status data of computing nodes, allocating data segments to corresponding computing nodes to generate task distribution mapping; starting gene analysis tasks using a parallel execution mechanism, monitoring execution progress, and generating real-time synchronization adjustment signals; migrating and adjusting tasks according to the adjustment signals, optimizing the task execution sequence, collecting gene analysis results from each computing node, and fusing key site information to obtain an integrated analysis dataset; validating the analysis dataset, and outputting the final millet gene screening results. This invention improves resource utilization and analysis efficiency during gene data processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the fields of bioinformatics and distributed computing technology, and in particular to a method, system and storage medium for screening millet genes based on data mining. Background Technology

[0002] With the rapid development of high-throughput sequencing technology, millet genomics research has entered the era of massive data processing. Traditional gene analysis methods typically employ static task allocation strategies, simply dividing gene sequences into equal-length segments for parallel computation. This method ignores the inherent density unevenness and regional heterogeneity of gene data, leading to a severe imbalance in the load on computing nodes.

[0003] While distributed computing frameworks have improved processing power to some extent, their inherent resource allocation mechanisms are ill-suited to the computationally intensive nature of highly variable regions in genomic data, resulting in low cluster resource utilization. Furthermore, existing methods lack effective runtime monitoring and dynamic adjustment capabilities. When data hotspots or node performance fluctuations occur, the system cannot respond promptly, severely limiting overall analysis efficiency.

[0004] Furthermore, in a distributed environment, the gene locus information output by multiple computing nodes often suffers from coordinate discrepancies and type conflicts. Traditional methods lack effective data consistency assurance mechanisms, directly impacting the accuracy of the final screening results. These technical bottlenecks severely restrict the large-scale and precise development of millet gene screening, urgently requiring a novel solution that comprehensively considers data characteristics, resource allocation, and result quality. Summary of the Invention

[0005] To address the aforementioned technical issues, this application provides a data mining-based method, system, and storage medium for screening millet genes, which improves resource utilization and analysis efficiency during gene data processing.

[0006] In a first aspect, this application provides a millet gene screening method based on data mining, the method comprising: Step S1: Obtain millet gene data and perform initial clustering processing on the millet gene data to divide the millet gene data into multiple data segments to obtain a segment set; Step S2: Evaluate the resource requirement indicators of each data segment in the segment set to determine the task allocation requirements; Step S3: Obtain the current load status data of each computing node, allocate the data segment to the corresponding computing node, and generate a task distribution mapping; Step S4: Start the gene analysis task on the computing node using a parallel execution mechanism, and monitor the execution progress of each computing node. When there is a difference between the execution progress and the preset time threshold, generate a real-time synchronization adjustment signal. Step S5: Based on the real-time synchronization adjustment signal, the task is migrated and adjusted, the task execution sequence is optimized, the gene analysis results of each computing node are collected from the optimized task execution sequence, and the key site information is fused to obtain the integrated analysis dataset; Step S6: Perform validation processing on the analyzed dataset and output the final millet gene screening results.

[0007] Secondly, this application provides a millet gene screening system based on data mining, the system comprising: The segmentation unit is used to acquire millet gene data and perform initial clustering processing on the millet gene data to divide the millet gene data into multiple data segments to obtain a segment set. The indicator evaluation unit is used to evaluate the resource requirement indicators of each data segment in the segment set and determine the task allocation requirements. The task distribution unit is used to obtain the current load status data of each computing node, allocate the data segment to the corresponding computing node, and generate a task distribution mapping. The signal adjustment unit is used to start the gene analysis task on the computing node using a parallel execution mechanism and monitor the execution progress of each computing node. When there is a difference between the execution progress and the preset time threshold, a real-time synchronization adjustment signal is generated. The data integration unit is used to migrate and adjust the task according to the real-time synchronization adjustment signal, optimize the task execution sequence, collect the gene analysis results of each computing node from the optimized task execution sequence, fuse key site information, and obtain the integrated analysis dataset. The gene screening unit is used to validate the analysis dataset and output the final millet gene screening results.

[0008] A third aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the above-described data mining-based millet gene screening method.

[0009] Compared with the prior art, the beneficial effects of the present invention are at least as follows: 1. In terms of resource utilization, the system achieves precise matching of computing resources through intelligent clustering based on density features and dynamic load assessment mechanism. The dual load balancing strategy of initial clustering and dynamic migration is adopted, which significantly improves the utilization rate of cluster resources, effectively solves the problem of uneven node activity caused by data skew, and significantly reduces computing costs.

[0010] 2. In terms of analysis efficiency, a dynamic optimization system for the entire process from data partitioning to result verification has been established. Through a closed-loop control mechanism of real-time monitoring and migration adjustment, the time deviation during task execution is kept to a minimum, and the overall analysis cycle is significantly shortened. In particular, when processing large-scale data of millions of gene loci, it can maintain a stable high throughput.

[0011] 3. Regarding the reliability of the results, a multi-level quality assurance mechanism was constructed. By using site association matrix fusion and majority voting conflict resolution algorithms, the data consistency in distributed computing was improved. Combined with the dual verification of synchronization and integrity, the accuracy and reproducibility of the final screening results were ensured, providing reliable data support for the study of millet gene function. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating the steps of the millet gene screening method based on data mining in the embodiments of this application; Figure 2 This is a schematic diagram of the feature space distribution of the K-means clustering results of millet gene data in the embodiments of this application; Figure 3 This is a radar chart showing the performance indicators of the millet gene screening system in the embodiments of this application; Figure 4 This is a structural diagram of the millet gene screening system based on data mining in the embodiments of this application. Detailed Implementation

[0014] This application provides a method, system, and storage medium for millet gene screening based on data mining. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. Example 1

[0015] Traditional methods cannot handle the significant variations in density and length within millet genome data, resulting in uneven workload distribution across computing nodes and overall speed being hampered by the slowest tasks. This application addresses this issue by employing a dynamic task scheduling and load balancing mechanism. First, it intelligently clusters millet genome data into uniform segments and quantifies their computational requirements. Then, it performs initial allocation and dynamic migration during runtime based on real-time node load, forming a "monitoring-adjustment" closed loop. This ensures the consistency of distributed results, ultimately achieving efficient and accurate whole-genome analysis, significantly improving resource utilization and analysis speed.

[0016] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 The data mining-based millet gene screening method in this application includes: Step S1: Obtain millet gene data and perform initial clustering processing on the millet gene data to divide the millet gene data into multiple data segments, thus obtaining a segment set.

[0017] The initial clustering process for millet gene data includes: Step S11: Processing the millet gene data using a preset clustering algorithm, dividing the millet gene data into multiple uniform data segments based on data density and length characteristics; Step S12: Preliminarily classifying the data segments, generating a segment set containing multiple data segments, extracting feature parameters for each data segment in the segment set, and recording the correspondence between feature parameters and segment identifiers, and verifying the segment set using feature parameters; Step S13: Determining whether the segment set meets the uniformity condition. If it does, the segment set is used as input data for resource demand assessment; if it does not, the clustering parameters are adjusted and the clustering process is repeated; Step S14: Repeating Step S13 until a segment set that meets the uniformity condition is obtained. The uniformity condition refers to the distribution uniformity of data segments based on gene density and length characteristics. It is determined by calculating the variance of the density mean and comparing it with a preset threshold, such as 0.05. If the variance is less than the threshold, it is considered to meet the uniformity condition.

[0018] Specifically, to address the issue of unbalanced cluster computing load caused by uneven data distribution in millet gene data processing, the initial clustering stage involves first acquiring millet gene data and then processing it using the K-means clustering algorithm, such as... Figure 2 The diagram shows the K-means clustering result and feature space distribution of millet gene data. The K-means clustering algorithm uses a feature vector composed of the gene density (number of gene loci per unit length) and sequence length of each data point as input to divide the data into K clusters, forming multiple initial data segments. Subsequently, feature extraction is performed on the generated segment set, specifically calculating the mean density, total length, and locus variability for each data segment. These feature parameters and their corresponding segment identifiers are recorded in a mapping table. Then, the uniformity of the segment set is checked based on the variance of the mean density. For example, a variance threshold of 0.05 is set. When the actual variance exceeds this threshold, the number of clusters K is increased (e.g., from the initial value of 5 to 8) or a length weight factor is introduced (e.g., gradually adjusted from 1.0 to 1.2), and clustering is re-executed. Through multiple iterations, the variance is reduced below the threshold, thus obtaining a segment set that meets the uniformity condition. Finally, this set is serialized and stored as the initial processing result, providing input for subsequent resource demand assessment based on dynamic programming algorithms.

[0019] By combining the above-mentioned dynamic adjustment of clustering parameters with the uniformity verification mechanism, the data segments are balanced in terms of density and length, thus avoiding data skew from the source and improving the overall efficiency of the millet gene screening task.

[0020] Step S2: Evaluate the resource requirement indicators of each data segment in the segment set to determine the task allocation requirements.

[0021] Step S2 further includes: calculating the computational complexity of each data segment in the segment set using a dynamic programming algorithm; determining the load weight value of each data segment based on the computational complexity and the difference in millet gene density within the data segment; generating resource demand indicators for the corresponding data segment based on the load weight values; sorting the resource demand indicators to generate a resource demand priority list; marking high-complexity data segments and low-complexity data segments according to the resource demand priority list; comparing the resource demand indicators of high-complexity data segments with preset indicator thresholds to determine whether resources need to be allocated preferentially; marking the data segment as a high-priority task if the resource demand indicators exceed the preset indicator thresholds; updating the resource demand indicator allocation strategy based on the marking results, and using the updated resource demand indicators as the basis for task allocation.

[0022] Specifically, to address the issue of inaccurate resource allocation in gene data analysis caused by the inability to precisely quantify the actual computational load of each segment, the resource requirement assessment phase first employs a dynamic programming algorithm to calculate the computational complexity of each segment. Dynamic programming is an optimization algorithm that decomposes a complex problem into interconnected subproblems and solves them in a bottom-up manner. Specifically, each data segment containing a specific number of gene loci, such as 2000 loci, is decomposed into continuous subsequences. A dynamic programming state table is constructed, with the end position of the subsequence as the state and sequence length and gene density as the state transition criteria. The local computational load of each subsequence is recursively calculated and accumulated to obtain the overall computational complexity, thus outputting a complexity value measured in computational units, such as 800 units. Subsequently, considering the differences in gene density within the segment, the complexity is calculated per 100 gene units. The system quantifies the standard deviation of the number of sites, for example, with a standard deviation of 5. It then weights the complexity by multiplying the complexity value by the density standard deviation to obtain the load weight value. This load weight value is then linearly normalized to map to a range of 0 to 1, generating a resource requirement index, for example, 0.8. Next, the resource requirement indices of all segments are sorted in descending order to generate a priority list. For example, the top 30% of segments are marked as high-complexity segments, and their resource requirement indices are compared with a preset threshold, such as 0.7. If the index exceeds the threshold, such as 0.85, it is marked as a high-priority task. Finally, the allocation strategy is updated based on the marking results. For example, the resource requirement index of high-priority tasks is multiplied by a weighting coefficient of 1.5, or historical execution data is used to allocate them to high-performance nodes with a load below 50%, thus forming an updated resource requirement index as the direct basis for subsequent task allocation.

[0023] By combining dynamic programming with density difference weighting in the above-mentioned evaluation mechanism, multi-dimensional and accurate quantification of gene data computational load is achieved, providing a reliable data foundation for dynamic task scheduling and thus improving task scheduling efficiency.

[0024] Step S3: Obtain the current load status data of each computing node, allocate the data segments to the corresponding computing nodes, and generate a task distribution map.

[0025] The process of allocating data segments to corresponding computing nodes includes: obtaining the current load status data of each computing node and comparing the load status data of each computing node with a preset load threshold. If the load status data of a computing node is lower than the preset load threshold, the computing node is marked as an available node. Based on resource requirement indicators, high-complexity data segments are preferentially allocated to available nodes. Through the allocation process, an initial task distribution map is generated. The task distribution map is then subjected to load balancing verification to determine whether the load of each computing node is balanced. If it is not balanced, the data segment allocation scheme in the task distribution map is adjusted until a task distribution map that meets the load balancing conditions is obtained. The task distribution map is then used as the basic configuration for parallel task execution.

[0026] Specifically, to address the issues of task execution delays and low resource utilization caused by uneven computing node loads in gene data processing, during the task allocation phase, the CPU utilization, memory usage, and network bandwidth metrics of each node in the distributed computing cluster are first queried to form a load status dataset. Each node's metrics are then compared to a preset load threshold, such as 80% CPU utilization. Nodes with load status data below this preset threshold are marked as available nodes. Subsequently, based on the load weight values ​​obtained from the resource demand assessment phase (calculated using a dynamic programming algorithm and gene density differences), data segments are sorted in descending order of load weight values. The segments with the highest weight values ​​are preferentially allocated to the available nodes with the lowest current load. This allocation process is repeated until all segments are allocated, generating an initial task distribution map. Finally, the initial map is further processed... Load balancing verification is performed by calculating the sum of the total load weights of the segments allocated to each node as the node's total load index, and then calculating the variance of these total load indices. This variance is compared with a preset load balancing threshold; if the variance exceeds the threshold, the node is considered unbalanced. For unbalanced nodes, a greedy algorithm is used for dynamic adjustment. This involves migrating data segments from high-load nodes to low-load nodes that minimize variance. The greedy algorithm, which makes the best decision at each step, is used in this scheme during the task migration adjustment phase. By selecting data segments that minimize cluster load variance at each migration step, rapid convergence of load balancing is achieved. After dynamic adjustment, the task distribution mapping is updated. Through iterative adjustments, the variance is reduced to below the threshold, ultimately obtaining a task distribution mapping that meets the load balancing conditions. This mapping serves as the basic configuration for subsequent parallel execution of gene analysis tasks.

[0027] Through the dynamic allocation and verification mechanism based on real-time load status and greedy algorithm, the accurate matching of computing resources and the uniform distribution of task load are achieved, which effectively improves the overall processing efficiency of the cluster and ensures the efficient parallel execution of the millet gene screening process.

[0028] Step S4: Start the gene analysis task on the computing node using a parallel execution mechanism and monitor the execution progress of each computing node. When there is a difference between the execution progress and the preset time threshold, generate a real-time synchronization adjustment signal.

[0029] Step S4 further includes: initializing gene analysis tasks on multiple computing nodes according to the task distribution mapping, and starting the processing flow of gene analysis tasks through a parallel execution mechanism; monitoring the execution time progress of gene analysis tasks on each computing node in real time, comparing the execution time progress with a preset time threshold to determine if there is a time difference, and if so, generating a real-time synchronization adjustment signal; otherwise, continuing to monitor the execution time progress of gene analysis tasks; marking the computing nodes and task segments that need to be adjusted through the real-time synchronization adjustment signal, recording the marking information as the basis for adjustment, and transmitting the real-time synchronization adjustment signal to the task scheduling module.

[0030] Specifically, to address the overall efficiency degradation caused by asynchronous node execution progress during parallel gene analysis, in the task execution and monitoring phase, the corresponding gene data segments are first loaded onto each computing node according to the task distribution mapping, and memory and thread parameters are configured. Parallel analysis tasks are then synchronously started via a message passing interface. Subsequently, a real-time monitoring mechanism is established to collect the percentage of gene loci completed by each node at fixed time intervals, i.e., the current processing progress. The cumulative execution time is calculated by combining the expected time model built based on the load weight value, and the actual progress is compared with a preset time threshold. When the execution progress is detected to be lagging behind the preset time threshold, a real-time synchronization adjustment signal is generated, which includes node identification and difference information. Then, based on the signal analysis, the segment that needs to be adjusted is determined, and the migration amount is calculated by combining gene density and length characteristics. Finally, the marker information is transmitted to the scheduling module via a network protocol to trigger the migration operation, forming a complete closed-loop control from progress monitoring to dynamic adjustment. The expected time model is based on historical gene analysis task data, with the load weight value of the data segment as input. A statistical model for estimating execution time is trained through linear regression, where the load weight value is calculated by a dynamic programming algorithm and gene density differences.

[0031] Step S5: Based on the real-time synchronization adjustment signal, the task is migrated and adjusted, the task execution sequence is optimized, the gene analysis results of each computing node are collected from the optimized task execution sequence, and the key site information is fused to obtain the integrated analysis dataset.

[0032] The task migration and adjustment process includes: receiving and parsing the task segments marked in the real-time synchronization adjustment signal that need to be adjusted, identifying the current computing node and execution status of the task segment; determining the remaining computing load of the task segment based on the execution status, finding the computing node with the lowest current load based on the remaining computing load, and using a task migration protocol to transfer some of the computing tasks of the task segment to the computing node with the lowest load; updating the task distribution mapping through task transfer, and predicting the execution time of the updated task distribution mapping to determine whether the overall execution time is balanced. If it is not balanced, the task allocation is adjusted again. After multiple adjustments, an optimized task execution sequence is generated and used as the final task scheduling scheme.

[0033] Specifically, to address the issues of decreased cluster resource utilization and overall analysis delays caused by imbalanced task execution progress during gene data analysis, the migration and adjustment phase first receives and parses the task segments requiring adjustment marked in the real-time synchronization adjustment signal. It then identifies the current computing node and execution status by querying node logs, and calculates the remaining computational load based on the proportion of completed tasks (e.g., remaining computational load = total computational load - processed load). Subsequently, based on the remaining computational load, it scans the node list to select available nodes with loads below average, and uses a task migration protocol to transfer some computational tasks to the target nodes. This migration protocol involves dividing the data into migrateable subtasks, such as separating high-density gene data with a second computational load from a segment with a first computational load, performing data replication, and synchronizing the status. The computational load is less than the initial computational load, and integrity verification after the migration ensures lossless data transmission. After the migration is completed, the task distribution mapping is updated, and the execution time of the updated mapping is predicted by a linear regression prediction model built based on historical execution data. This linear regression prediction model takes node load and segment complexity as inputs and outputs the expected execution time of each node. For example, if node A is predicted to be 10 minutes and node B is predicted to be 8 minutes, the average of the predicted times of all nodes is calculated and compared with the maximum value. If the difference exceeds 10%, it is determined to be unbalanced. When there is an imbalance, the above migration and prediction process is repeated. After multiple iterations, the overall time deviation is controlled within 5%. Finally, an optimized task execution sequence is generated as the final scheduling scheme for subsequent analysis, thereby achieving continuous optimization of the execution sequence and improving resource utilization and analysis speed.

[0034] The process of collecting gene analysis results from each computing node from the optimized task execution sequence includes: obtaining the gene analysis task results completed by each computing node according to the optimized task execution sequence; extracting key site information from the gene analysis task results; performing data cleaning and format unification processing using the key site information; fusing the processed key site information to generate a preliminary integrated dataset; performing consistency verification on the preliminary integrated dataset to determine if there are any data conflicts; if so, correcting the conflicting data; if not, determining the preliminary integrated dataset as the integrated analysis dataset; and recording the distribution characteristics of each key site based on the analysis dataset, using the distribution characteristics as input data for verification processing.

[0035] Specifically, to address the reliability issue of data integration caused by inconsistent outputs from parallel computing nodes in distributed gene analysis, the following steps are taken during the result collection and integration phase: First, task completion markers for each computing node are extracted based on the optimized task execution sequence. Gene analysis task results are then retrieved from local storage, and variant site sequence fragments are identified to extract site coordinates and type information. Next, key site information undergoes data cleaning, with noise data removed using filtering rules and converted to a standardized format. A merge sort algorithm is then used to sort and merge data by site coordinates, eliminating duplicates, and constructing a site association matrix. The matrix elements represent the co-occurrence frequency of sites within the millet genome, generating a preliminary integrated dataset. Finally, the preliminary integrated data is processed... The dataset undergoes consistency verification by traversing each site and comparing attribute values ​​from different nodes. If the coordinate deviation exceeds a preset threshold, it is marked as a data conflict, and the number of conflicts is counted to determine if inconsistency exists. If a conflict exists, a majority voting mechanism is applied for correction. The original results of all relevant nodes are collected, and the frequency of each attribute value is counted. Values ​​with a frequency exceeding 50% are selected, with the threshold adjusted to 60% for sparse gene segments, as correction values ​​to update the dataset. This process is iterated until there are no conflicts. Finally, the conflict-free dataset is determined as the integrated analysis dataset. The density and spacing distribution of key sites are calculated as distribution features, packaged into structured data packets, and transmitted to the verification module to provide input for subsequent synchronization verification.

[0036] The multi-node result fusion and automatic conflict correction mechanism effectively eliminates data ambiguity in distributed computing, ensures the integrity and reliability of whole genome analysis data, and thus ensures the accuracy and consistency of millet gene screening results.

[0037] Step S6: Perform validation processing on the analyzed dataset and output the final millet gene screening results.

[0038] Step S6 further includes: acquiring and extracting synchronization index data from the integrated analysis dataset; comparing the synchronization index data with preset synchronization requirements; if the synchronization index data meets the preset synchronization requirements, marking the analysis dataset as a qualified dataset; otherwise, correcting the analysis dataset; updating the synchronization index data and re-comparing it after correcting the analysis dataset; performing a final data integrity check on the qualified dataset and generating a verification result report; determining whether the analysis dataset can be used as the final output based on the verification result report; and storing the verified analysis dataset as the final gene screening result.

[0039] Specifically, to address the reliability issues of final screening results caused by asynchronous task execution and incomplete data in gene data analysis, the validation process first extracts synchronization index data from the integrated whole-genome analysis dataset. Synchronization is quantified by calculating the standard deviation of execution time for each data segment and comparing this value with preset synchronization requirements. If the standard deviation is less than a threshold, the dataset is directly marked as a qualified dataset; otherwise, a correction process is initiated. This involves identifying high-complexity segments, such as those with gene density greater than twice the average, and recalculating their load weight values ​​using a dynamic programming algorithm (e.g., load weight = density × length / average). Then, a task migration protocol is used to transfer some data subtasks to nodes with loads below a preset threshold. After updating the execution time progress, the synchronization index data is recalculated and iteratively compared until the requirements are met. Subsequently, a data integrity check is performed on the qualified dataset to verify the completeness and completeness of key site information, generating a validation result report containing the number of sites and their missing information. Finally, based on the report, the dataset is confirmed as output and stored as the millet gene screening results.

[0040] By combining the above-mentioned synchronization verification and dynamic correction mechanisms, the risks of time deviation and data loss in the parallel analysis process are effectively eliminated, ensuring the consistency and accuracy of gene screening results.

[0041] Through the coordination of the above steps, this application achieves dynamic optimization and precise control of the entire millet gene data processing chain, significantly improving the resource utilization and analysis efficiency of large-scale parallel computing.

[0042] In summary, this application achieves systematic optimization of the millet gene screening process by constructing a complete technical chain from data preprocessing to result verification. Specifically, it establishes a precise load quantification system by employing an intelligent clustering algorithm based on density and length features, combined with a dynamic programming resource demand assessment mechanism; it forms a dynamic-static load balancing scheme through real-time node status monitoring and task migration driven by a greedy algorithm; it solves the distributed result consistency problem by utilizing a site association matrix and a majority voting mechanism; and it ensures output quality through synchronization verification and integrity checks. This scheme, through the synergistic effect of each stage, achieves significant improvements in data processing efficiency, resource utilization, and result accuracy, providing reliable technical support for large-scale genome analysis. The technical effects achievable by this application are as follows: Figure 3 The radar chart showing the performance metrics of the millet gene screening system is shown. Example 2

[0043] The data mining-based millet gene screening method in the embodiments of this application has been described above. The data mining-based millet gene screening system in the embodiments of this application is described below. Please refer to [link to relevant documentation]. Figure 4 The data mining-based millet gene screening system in this application includes: The segmentation unit is used to acquire millet gene data and perform initial clustering processing on the millet gene data to divide the millet gene data into multiple data segments, thus obtaining a segment set. The indicator evaluation unit is used to evaluate the resource requirement indicators of each data segment in the segment set and determine the task allocation requirements. The task distribution unit is used to obtain the current load status data of each computing node, allocate data segments to the corresponding computing nodes, and generate task distribution mappings. The signal adjustment unit is used to start gene analysis tasks on computing nodes using a parallel execution mechanism and monitor the execution progress of each computing node. When there is a difference between the execution progress and the preset time threshold, a real-time synchronization adjustment signal is generated. The data integration unit is used to migrate and adjust tasks based on real-time synchronization adjustment signals, optimize task execution sequences, collect gene analysis results from each computing node from the optimized task execution sequences, and fuse key site information to obtain an integrated analysis dataset. The gene screening unit is used to validate the analyzed dataset and output the final millet gene screening results.

[0044] Through the synergistic cooperation of the above components, the resource utilization and analysis efficiency of large-scale parallel computing are further improved. Example 3

[0045] This application also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, storing instructions that, when executed on a computer, cause the computer to perform the steps of a millet gene screening method based on data mining.

[0046] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0047] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0048] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data mining-based method for screening millet genes, characterized in that, The methods include: Step S1: Obtain millet gene data, and perform initial clustering processing on the millet gene data using the K-means clustering algorithm to divide the millet gene data into multiple data segments, obtaining a segment set; Step S2: Evaluate the resource requirement index of each data segment in the segment set to determine the task allocation requirements; Step S2 further includes: using a dynamic programming algorithm to calculate the computational complexity of each data segment in the segment set; based on the computational complexity and the difference in millet gene density within the data segment, determine the load weight value of each data segment, and generate corresponding data based on the load weight value. The system identifies resource requirement indicators for data segments; sorts these indicators to generate a resource requirement priority list; and marks high-complexity and low-complexity data segments based on this list. The system then compares the resource requirement indicators of the high-complexity data segments with preset threshold values ​​to determine if priority resource allocation is needed. If the resource requirement indicators exceed the preset threshold values, the data segment is marked as a high-priority task. Based on the marking results, the allocation strategy for the resource requirement indicators is updated, and the updated indicators are used as the basis for task allocation. Step S3: Obtain the current load status data of each computing node, allocate data segments to the corresponding computing nodes, and generate a task distribution map; Step S3, allocating the data segments to the corresponding computing nodes, includes: obtaining the current load status data of each computing node, comparing the load status data of each computing node with a preset load threshold, and marking the computing node as an available node if the load status data of a certain computing node is lower than the preset load threshold; according to the resource requirement index, preferentially allocating the high-complexity data segments to the available nodes, generating an initial task distribution map through the allocation process; performing load balancing verification on the task distribution map to determine whether the load of each computing node is balanced, and if not, adjusting the data segment allocation scheme in the task distribution map until a task distribution map that meets the load balancing conditions is obtained, and using the task distribution map as the basic configuration for parallel task execution; Step S4: Start the gene analysis task on the computing nodes using a parallel execution mechanism and monitor the execution progress of each computing node. When there is a difference between the execution progress and the preset time threshold, generate a real-time synchronization adjustment signal. Step S5: Based on the real-time synchronization adjustment signal, migrate and adjust the task, optimize the task execution sequence, collect the gene analysis results of each computing node from the optimized task execution sequence, and fuse key site information to obtain the integrated analysis dataset. Step S6: Validate the analysis dataset and output the final millet gene screening results.

2. The method of claim 1, wherein, Step S1 involves initial clustering of the millet gene data, including: Step S11: Process the millet gene data using a preset clustering algorithm, based on data density and length characteristics. The method divides the millet gene data into multiple uniform data segments; Step S12: Perform preliminary classification of data segments to generate a segment set containing multiple data segments, extract feature parameters of each data segment in the segment set, record the correspondence between feature parameters and segment identifiers, and verify the segment set through feature parameters; Step S13: Determine whether the segment set meets the uniformity condition. If it does, use the segment set as input data for resource demand assessment. If it does not, adjust the clustering parameters and re-cluster the data. Step S14: Repeat step S13 until a set of segments that satisfy the uniformity condition is obtained.

3. The method of claim 1, wherein, Step S4 also includes: Based on the task distribution mapping, initialize gene analysis tasks on multiple computing nodes, and start the processing flow of gene analysis tasks through a parallel execution mechanism; The execution time progress of gene analysis tasks on each computing node is monitored in real time. The execution time progress is compared with a preset time threshold to determine if there is a time difference. If there is, a real-time synchronization adjustment signal is generated; otherwise, the execution time progress of gene analysis tasks continues to be monitored.

4. The method of claim 1, wherein, Step S5 involves migrating and adjusting the task, including: Receive and parse the task segments that need adjustment marked in the real-time synchronization adjustment signal, and identify the current computing node and execution status of the task segments; Based on the execution status, the remaining computing power of the task segment is determined. According to the remaining computing power, a computing node with a lower current load is found. A task migration protocol is used to transfer some of the computing tasks of the task segment to the computing node with a lower load. By transferring tasks, the task distribution mapping is updated, and the execution time of the updated task distribution mapping is predicted to determine whether the overall execution time is balanced. If it is not balanced, the task allocation is adjusted again. After multiple adjustments, an optimized task execution sequence is generated and used as the final task scheduling scheme.

5. The method of claim 4, wherein, Step S5 involves collecting the gene analysis results from each computing node in the optimization task execution sequence, including: According to the optimized task execution sequence, the gene analysis task results completed by each computing node are obtained, key site information in the gene analysis task results is extracted, data cleaning and format unification processing are performed through the key site information, and the processed key site information is fused to generate a preliminary integrated dataset. A consistency check is performed on the preliminary integrated dataset to determine whether there are any data conflicts. If there are, the conflicting data is corrected; if not, the preliminary integrated dataset is determined as the integrated analysis dataset.

6. The method of claim 1, wherein, Step S6 further includes: Acquire and extract synchronization index data from the integrated analysis dataset. Based on the synchronization index data, compare it with the preset synchronization requirements. If the synchronization index data meets the preset synchronization requirements, mark the analysis dataset as a qualified dataset. Otherwise, correct the analysis dataset. After correcting the analysis dataset, update the synchronization index data and compare it again. For the qualified dataset, perform a final data integrity check and generate a verification result report. Based on the verification result report, determine whether the analysis dataset can be used as the final output, and store the verified analysis dataset as the final gene screening result.

7. A data mining-based millet gene screening system for implementing the data mining-based millet gene screening method according to any one of claims 1 to 6, characterized in that, The system includes: The segmentation unit is used to acquire millet gene data and perform initial clustering processing on the millet gene data to divide the millet gene data into multiple data segments, thus obtaining a segment set. The indicator evaluation unit is used to evaluate the resource requirement indicators of each data segment in the segment set and determine the task allocation requirements. The task distribution unit is used to obtain the current load status data of each computing node, allocate data segments to the corresponding computing nodes, and generate task distribution mappings. The signal adjustment unit is used to start gene analysis tasks on computing nodes using a parallel execution mechanism and monitor the execution progress of each computing node. When there is a difference between the execution progress and the preset time threshold, a real-time synchronization adjustment signal is generated. The data integration unit is used to migrate and adjust tasks based on real-time synchronization adjustment signals, optimize task execution sequences, collect gene analysis results from each computing node from the optimized task execution sequences, and fuse key site information to obtain an integrated analysis dataset. The gene screening unit is used to validate the analyzed dataset and output the final millet gene screening results.

8. A computer-readable storage medium having stored thereon instructions, the computer-readable storage medium comprising: When the instructions are executed by the processor, they implement the millet gene screening method based on data mining as described in any one of claims 1-6.

Citation Information

Patent Citations

  • Self-adaptive task scheduling system and method

    CN120144257A

  • Cloud side-end cooperative control method and system based on remote communication

    CN121262210A

  • Method and system for facilitating optimization of a cluster computing network for sequencing data analysis using adaptive data parallelization, and non-transitory storage medium

    US20220157414A1