Setaria italica gene screening method and system based on data mining and storage medium
By using data mining-based methods, intelligent clustering and dynamic load balancing of millet genetic data were achieved, solving the problems of unbalanced computing node load and data consistency, improving resource utilization and analysis efficiency, and ensuring the accuracy and reliability of the results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ANYANG ACAD OF AGRI SCI
- Filing Date
- 2026-01-22
- Publication Date
- 2026-05-01
AI Technical Summary
Existing technologies in millet genomics research suffer from problems such as uneven computing node load, low resource utilization, and poor data consistency, which limit the efficiency of analysis and the accuracy of results.
By employing data mining-based methods, intelligent clustering based on density features and dynamic load assessment, combined with dynamic migration adjustments, we can achieve precise matching of computing resources and real-time monitoring of task execution. This enables the construction of a multi-level quality assurance mechanism to ensure data consistency and result accuracy in distributed computing.
It significantly improved the utilization rate of cluster resources, shortened the analysis cycle, maintained high throughput, and ensured the accuracy and reliability of the final screening results, providing reliable data support for the study of millet gene function.
Smart Images
Figure CN121963847A_ABST
Abstract
Description
A data mining-based method, system, and storage medium for millet gene screening. Technical Field
[0001] This application relates to the fields of bioinformatics and distributed computing technology, and in particular to a method, system and storage medium for screening millet genes based on data mining. Background Technology
[0002] With the rapid development of high-throughput sequencing technology, millet genomics research has entered the era of massive data processing. Traditional gene analysis methods typically employ static task allocation strategies, simply dividing gene sequences into equal-length segments for parallel computation. This method ignores the inherent density unevenness and regional heterogeneity of gene data, leading to a severe imbalance in the load on computing nodes.
[0003] While distributed computing frameworks have improved processing power to some extent, their inherent resource allocation mechanisms are ill-suited to the computationally intensive nature of highly variable regions in genomic data, resulting in low cluster resource utilization. Furthermore, existing methods lack effective runtime monitoring and dynamic adjustment capabilities. When data hotspots or node performance fluctuations occur, the system cannot respond promptly, severely limiting overall analysis efficiency.
[0004] Furthermore, in a distributed environment, the gene locus information output by multiple computing nodes often suffers from coordinate discrepancies and type conflicts. Traditional methods lack effective data consistency assurance mechanisms, directly impacting the accuracy of the final screening results. These technical bottlenecks severely restrict the large-scale and precise development of millet gene screening, urgently requiring a novel solution that comprehensively considers data characteristics, resource allocation, and result quality. Summary of the Invention
[0005] To address the aforementioned technical issues, this application provides a data mining-based method, system, and storage medium for screening millet genes, which improves resource utilization and analysis efficiency during gene data processing.
[0006] In a first aspect, this application provides a data mining-based method for millet gene screening, the method comprising: Step S1: acquiring millet gene data and performing initial clustering processing on the millet gene data to divide the millet gene data into multiple data segments, obtaining a segment set; Step S2: evaluating the resource requirement index of each data segment in the segment set to determine task allocation requirements; Step S3: acquiring the current load status data of each computing node, allocating the data segments to the corresponding computing nodes, and generating a task distribution mapping; Step S4: starting gene analysis tasks on the computing nodes using a parallel execution mechanism and monitoring the execution progress of each computing node. When the execution progress differs from a preset time threshold, a real-time synchronization adjustment signal is generated; Step S5: adjusting the tasks according to the real-time synchronization adjustment signal, optimizing the task execution sequence, collecting gene analysis results from each computing node from the optimized task execution sequence, fusing key site information, and obtaining an integrated analysis dataset; Step S6: validating the analysis dataset and outputting the final millet gene screening results.
[0007] Secondly, this application provides a millet gene screening system based on data mining. The system includes: a segmentation unit for acquiring millet gene data and performing initial clustering processing on the millet gene data to divide the millet gene data into multiple data segments, obtaining a segment set; an indicator evaluation unit for evaluating the resource requirement indicators of each data segment in the segment set and determining task allocation requirements; a task distribution unit for acquiring the current load status data of each computing node, allocating the data segments to the corresponding computing nodes, and generating a task distribution mapping; a signal adjustment unit for starting gene analysis tasks on the computing nodes using a parallel execution mechanism and monitoring the execution progress of each computing node, generating a real-time synchronization adjustment signal when the execution progress differs from a preset time threshold; a data integration unit for migrating and adjusting tasks according to the real-time synchronization adjustment signal, optimizing the task execution sequence, collecting gene analysis results from each computing node from the optimized task execution sequence, fusing key site information, and obtaining an integrated analysis dataset; and a gene screening unit for verifying the analysis dataset and outputting the final millet gene screening results.
[0008] A third aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the above-described data mining-based millet gene screening method.
[0009] Compared with the prior art, the beneficial effects of the present invention are at least as follows: 1. In terms of resource utilization, through the intelligent clustering and dynamic load assessment mechanism based on density features, the accurate matching of computing resources is achieved. The dual load balancing strategy combining initial clustering and dynamic migration is adopted, which significantly improves the utilization rate of cluster resources, effectively solves the problem of uneven node activity caused by data skew, and significantly reduces computing costs.
[0010] 2. In terms of analysis efficiency, a dynamic optimization system for the entire process from data partitioning to result verification has been established. Through a closed-loop control mechanism of real-time monitoring and migration adjustment, the time deviation during task execution is kept to a minimum, and the overall analysis cycle is significantly shortened. In particular, when processing large-scale data of millions of gene loci, it can maintain a stable high throughput.
[0011] 3. Regarding the reliability of the results, a multi-level quality assurance mechanism was constructed. By using site association matrix fusion and majority voting conflict resolution algorithms, the data consistency in distributed computing was improved. Combined with the dual verification of synchronization and integrity, the accuracy and reproducibility of the final screening results were ensured, providing reliable data support for the study of millet gene function. Attached Figure Description
[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 is a flowchart of the steps of the millet gene screening method based on data mining in this embodiment of the application; Figure 2 is a schematic diagram of the feature space distribution of the K-means clustering results of millet gene data in this embodiment of the application; Figure 3 is a radar chart of the performance index of the millet gene screening system in this embodiment of the application; Figure 4 is a structural diagram of the millet gene screening system based on data mining in this embodiment of the application. Detailed Implementation
[0014] This application provides a method, system, and storage medium for millet gene screening based on data mining. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data used can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. Example 1
[0015] Traditional methods cannot handle the significant variations in density and length within millet genome data, resulting in uneven workload distribution across computing nodes and overall speed being hampered by the slowest tasks. This application addresses this issue by employing a dynamic task scheduling and load balancing mechanism. First, it intelligently clusters millet genome data into uniform segments and quantifies their computational requirements. Then, it performs initial allocation and dynamic migration during runtime based on real-time node load, forming a "monitoring-adjustment" closed loop. This ensures the consistency of distributed results, ultimately achieving efficient and accurate whole-genome analysis, significantly improving resource utilization and analysis speed.
[0016] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to Figure 1. The millet gene screening method based on data mining in the embodiments of this application includes: Step S1: Obtain millet gene data and perform initial clustering processing on the millet gene data to divide the millet gene data into multiple data segments to obtain a segment set.
[0017] The initial clustering process for millet gene data includes: Step S11: Processing the millet gene data using a preset clustering algorithm, dividing the millet gene data into multiple uniform data segments based on data density and length characteristics; Step S12: Preliminarily classifying the data segments, generating a segment set containing multiple data segments, extracting feature parameters for each data segment in the segment set, and recording the correspondence between feature parameters and segment identifiers, and verifying the segment set using feature parameters; Step S13: Determining whether the segment set meets the uniformity condition. If it does, the segment set is used as input data for resource demand assessment; if it does not, the clustering parameters are adjusted and the clustering process is repeated; Step S14: Repeating Step S13 until a segment set that meets the uniformity condition is obtained. The uniformity condition refers to the distribution uniformity of data segments based on gene density and length characteristics. It is determined by calculating the variance of the density mean and comparing it with a preset threshold, such as 0.05. If the variance is less than the threshold, it is considered to meet the uniformity condition.
[0018] Specifically, to address the cluster computing load imbalance caused by uneven data distribution in millet gene data processing, the initial clustering stage involves first acquiring millet gene data and processing it using the K-means clustering algorithm. Figure 2 shows the K-means clustering result and feature space distribution diagram of the millet gene data. The K-means clustering algorithm uses the gene density of each data point (i.e., the number of gene loci per unit length and the sequence length) as input to form a feature vector, dividing the data into K clusters, thus creating multiple preliminary data segments. Subsequently, feature extraction is performed on the generated segment set, specifically calculating the density of each data segment. The mean density, total length, and site variability are calculated, and these characteristic parameters are recorded in a mapping table along with their corresponding segment identifiers. Then, the uniformity of the segment set is checked based on the variance of the mean density. For example, a variance threshold of 0.05 is set. When the actual variance exceeds this threshold, the number of clusters K is increased (e.g., from the initial value of 5 to 8) or a length weight factor is introduced (e.g., gradually adjusted from 1.0 to 1.2), and clustering is re-executed. Through multiple iterations, the variance is reduced below the threshold, thus obtaining a segment set that meets the uniformity condition. Finally, this set is serialized and stored as the initial processing result, providing input for subsequent resource demand assessment based on dynamic programming algorithms.
[0019] By combining the above-mentioned dynamic adjustment of clustering parameters with the uniformity verification mechanism, the data segments are balanced in terms of density and length, thus avoiding data skew from the source and improving the overall efficiency of the millet gene screening task.
[0020] Step S2: Evaluate the resource requirement indicators of each data segment in the segment set to determine the task allocation requirements.
[0021] Step S2 further includes: calculating the computational complexity of each data segment in the segment set using a dynamic programming algorithm; determining the load weight value of each data segment based on the computational complexity and the difference in millet gene density within the data segment; generating resource demand indicators for the corresponding data segment based on the load weight values; sorting the resource demand indicators to generate a resource demand priority list; marking high-complexity data segments and low-complexity data segments according to the resource demand priority list; comparing the resource demand indicators of high-complexity data segments with preset indicator thresholds to determine whether resources need to be allocated preferentially; marking the data segment as a high-priority task if the resource demand indicators exceed the preset indicator thresholds; updating the resource demand indicator allocation strategy based on the marking results, and using the updated resource demand indicators as the basis for task allocation.
[0022] Specifically, to address the issue of inaccurate resource allocation in gene data analysis caused by the inability to precisely quantify the actual computational load of each segment, the resource requirement assessment phase first employs a dynamic programming algorithm to calculate the computational complexity of each segment. Dynamic programming is an optimization algorithm that decomposes a complex problem into interconnected subproblems and solves them in a bottom-up manner. Specifically, each data segment containing a specific number of gene loci, such as 2000 loci, is decomposed into continuous subsequences. A dynamic programming state table is constructed, with the end position of the subsequence as the state and sequence length and gene density as the state transition criteria. The local computational load of each subsequence is recursively calculated and accumulated to obtain the overall computational complexity, thus outputting a complexity value measured in computational units, such as 800 units. Subsequently, considering the differences in gene density within the segment, the complexity is calculated per 100 gene units. The system quantifies the standard deviation of the number of sites, for example, with a standard deviation of 5. It then weights the complexity by multiplying the complexity value by the density standard deviation to obtain the load weight value. This load weight value is then linearly normalized to map to a range of 0 to 1, generating a resource requirement index, for example, 0.8. Next, the resource requirement indices of all segments are sorted in descending order to generate a priority list. For example, the top 30% of segments are marked as high-complexity segments, and their resource requirement indices are compared with a preset threshold, such as 0.7. If the index exceeds the threshold, such as 0.85, it is marked as a high-priority task. Finally, the allocation strategy is updated based on the marking results. For example, the resource requirement index of high-priority tasks is multiplied by a weighting coefficient of 1.5, or historical execution data is used to allocate them to high-performance nodes with a load below 50%, thus forming an updated resource requirement index as the direct basis for subsequent task allocation.
[0023] By combining dynamic programming with density difference weighting in the above-mentioned evaluation mechanism, multi-dimensional and accurate quantification of gene data computational load is achieved, providing a reliable data foundation for dynamic task scheduling and thus improving task scheduling efficiency.
[0024] Step S3: Obtain the current load status data of each computing node, allocate the data segments to the corresponding computing nodes, and generate a task distribution map.
[0025] The process of allocating data segments to corresponding computing nodes includes: obtaining the current load status data of each computing node and comparing the load status data of each computing node with a preset load threshold. If the load status data of a computing node is lower than the preset load threshold, the computing node is marked as an available node. Based on resource requirement indicators, high-complexity data segments are preferentially allocated to available nodes. Through the allocation process, an initial task distribution map is generated. The task distribution map is then subjected to load balancing verification to determine whether the load of each computing node is balanced. If it is not balanced, the data segment allocation scheme in the task distribution map is adjusted until a task distribution map that meets the load balancing conditions is obtained. The task distribution map is then used as the basic configuration for parallel task execution.
[0026] Specifically, to address the issues of task execution delays and low resource utilization caused by uneven computing node loads in gene data processing, during the task allocation phase, the CPU utilization, memory usage, and network bandwidth metrics of each node in the distributed computing cluster are first queried to form a load status dataset. Each node's metrics are then compared to a preset load threshold, such as 80% CPU utilization. Nodes with load status data below this preset threshold are marked as available nodes. Subsequently, based on the load weight values obtained from the resource demand assessment phase (calculated using a dynamic programming algorithm and gene density differences), data segments are sorted in descending order of load weight values. The segments with the highest weight values are preferentially allocated to the available nodes with the lowest current load. This allocation process is repeated until all segments are allocated, generating an initial task distribution map. Finally, the initial map is further processed... Load balancing verification is performed by calculating the sum of the total load weights of the segments allocated to each node as the node's total load index, and then calculating the variance of these total load indices. This variance is compared with a preset load balancing threshold; if the variance exceeds the threshold, the node is considered unbalanced. For unbalanced nodes, a greedy algorithm is used for dynamic adjustment. This involves migrating data segments from high-load nodes to low-load nodes that minimize variance. The greedy algorithm, which makes the best decision at each step, is used in this scheme during the task migration adjustment phase. By selecting data segments that minimize cluster load variance at each migration step, rapid convergence of load balancing is achieved. After dynamic adjustment, the task distribution mapping is updated. Through iterative adjustments, the variance is reduced to below the threshold, ultimately obtaining a task distribution mapping that meets the load balancing conditions. This mapping serves as the basic configuration for subsequent parallel execution of gene analysis tasks.
[0027] Through the dynamic allocation and verification mechanism based on real-time load status and greedy algorithm, the accurate matching of computing resources and the uniform distribution of task load are achieved, which effectively improves the overall processing efficiency of the cluster and ensures the efficient parallel execution of the millet gene screening process.
[0028] Step S4: Start the gene analysis task on the computing node using a parallel execution mechanism and monitor the execution progress of each computing node. When there is a difference between the execution progress and the preset time threshold, generate a real-time synchronization adjustment signal.
[0029] Step S4 further includes: initializing gene analysis tasks on multiple computing nodes according to the task distribution mapping, and starting the processing flow of gene analysis tasks through a parallel execution mechanism; monitoring the execution time progress of gene analysis tasks on each computing node in real time, comparing the execution time progress with a preset time threshold to determine if there is a time difference, and if so, generating a real-time synchronization adjustment signal; otherwise, continuing to monitor the execution time progress of gene analysis tasks; marking the computing nodes and task segments that need to be adjusted through the real-time synchronization adjustment signal, recording the marking information as the basis for adjustment, and transmitting the real-time synchronization adjustment signal to the task scheduling module.
[0030] Specifically, to address the overall efficiency degradation caused by asynchronous node execution progress during parallel gene analysis, in the task execution and monitoring phase, the corresponding gene data segments are first loaded onto each computing node according to the task distribution mapping, and memory and thread parameters are configured. Parallel analysis tasks are then synchronously started via a message passing interface. Subsequently, a real-time monitoring mechanism is established to collect the percentage of gene loci completed by each node at fixed time intervals, i.e., the current processing progress. The cumulative execution time is calculated by combining the expected time model built based on the load weight value, and the actual progress is compared with a preset time threshold. When the execution progress is detected to be lagging behind the preset time threshold, a real-time synchronization adjustment signal is generated, which includes node identification and difference information. Then, based on the signal analysis, the segment that needs to be adjusted is determined, and the migration amount is calculated by combining gene density and length characteristics. Finally, the marker information is transmitted to the scheduling module via a network protocol to trigger the migration operation, forming a complete closed-loop control from progress monitoring to dynamic adjustment. The expected time model is based on historical gene analysis task data, with the load weight value of the data segment as input. A statistical model for estimating execution time is trained through linear regression, where the load weight value is calculated by a dynamic programming algorithm and gene density differences.
[0031] Step S5: Based on the real-time synchronization adjustment signal, the task is migrated and adjusted, the task execution sequence is optimized, the gene analysis results of each computing node are collected from the optimized task execution sequence, and the key site information is fused to obtain the integrated analysis dataset.
[0032] The task migration and adjustment process includes: receiving and parsing the task segments marked in the real-time synchronization adjustment signal that need to be adjusted, identifying the current computing node and execution status of the task segment; determining the remaining computing load of the task segment based on the execution status, finding the computing node with the lowest current load based on the remaining computing load, and using a task migration protocol to transfer some of the computing tasks of the task segment to the computing node with the lowest load; updating the task distribution mapping through task transfer, and predicting the execution time of the updated task distribution mapping to determine whether the overall execution time is balanced. If it is not balanced, the task allocation is adjusted again. After multiple adjustments, an optimized task execution sequence is generated and used as the final task scheduling scheme.
[0033] Specifically, to address the issues of decreased cluster resource utilization and overall analysis delays caused by imbalanced task execution progress during gene data analysis, the migration and adjustment phase first receives and parses the task segments requiring adjustment marked in the real-time synchronization adjustment signal. It then identifies the current computing node and execution status by querying node logs, and calculates the remaining computational load based on the proportion of completed tasks (e.g., remaining computational load = total computational load - processed load). Subsequently, based on the remaining computational load, it scans the node list to select available nodes with loads below average, and uses a task migration protocol to transfer some computational tasks to the target nodes. This migration protocol involves dividing the data into migrateable subtasks, such as separating high-density gene data with a second computational load from a segment with a first computational load, performing data replication, and synchronizing the status. The computational load is less than the initial computational load, and integrity verification after the migration ensures lossless data transmission. After the migration is completed, the task distribution mapping is updated, and the execution time of the updated mapping is predicted by a linear regression prediction model built based on historical execution data. This linear regression prediction model takes node load and segment complexity as inputs and outputs the expected execution time of each node. For example, if node A is predicted to be 10 minutes and node B is predicted to be 8 minutes, the average of the predicted times of all nodes is calculated and compared with the maximum value. If the difference exceeds 10%, it is determined to be unbalanced. When there is an imbalance, the above migration and prediction process is repeated. After multiple iterations, the overall time deviation is controlled within 5%. Finally, an optimized task execution sequence is generated as the final scheduling scheme for subsequent analysis, thereby achieving continuous optimization of the execution sequence and improving resource utilization and analysis speed.
[0034] The process of collecting gene analysis results from each computing node from the optimized task execution sequence includes: obtaining the gene analysis task results completed by each computing node according to the optimized task execution sequence; extracting key site information from the gene analysis task results; performing data cleaning and format unification processing using the key site information; fusing the processed key site information to generate a preliminary integrated dataset; performing consistency verification on the preliminary integrated dataset to determine if there are any data conflicts; if so, correcting the conflicting data; if not, determining the preliminary integrated dataset as the integrated analysis dataset; and recording the distribution characteristics of each key site based on the analysis dataset, using the distribution characteristics as input data for verification processing.
[0035] Specifically, to address the reliability issue of data integration caused by inconsistent outputs from parallel computing nodes in distributed gene analysis, the following steps are taken during the result collection and integration phase: First, task completion markers for each computing node are extracted based on the optimized task execution sequence. Gene analysis task results are then retrieved from local storage, and variant site sequence fragments are identified to extract site coordinates and type information. Next, key site information undergoes data cleaning, with noise data removed using filtering rules and converted to a standardized format. A merge sort algorithm is then used to sort and merge data by site coordinates, eliminating duplicates, and constructing a site association matrix. The matrix elements represent the co-occurrence frequency of sites within the millet genome, generating a preliminary integrated dataset. Finally, the preliminary integrated data is processed... The dataset undergoes consistency verification by traversing each site and comparing attribute values from different nodes. If the coordinate deviation exceeds a preset threshold, it is marked as a data conflict, and the number of conflicts is counted to determine if inconsistency exists. If a conflict exists, a majority voting mechanism is applied for correction. The original results of all relevant nodes are collected, and the frequency of each attribute value is counted. Values with a frequency exceeding 50% are selected, with the threshold adjusted to 60% for sparse gene segments, as correction values to update the dataset. This process is iterated until there are no conflicts. Finally, the conflict-free dataset is determined as the integrated analysis dataset. The density and spacing distribution of key sites are calculated as distribution features, packaged into structured data packets, and transmitted to the verification module to provide input for subsequent synchronization verification.
[0036] The multi-node result fusion and automatic conflict correction mechanism effectively eliminates data ambiguity in distributed computing, ensures the integrity and reliability of whole genome analysis data, and thus ensures the accuracy and consistency of millet gene screening results.
[0037] Step S6: Perform validation processing on the analyzed dataset and output the final millet gene screening results.
[0038] Step S6 further includes: acquiring and extracting synchronization index data from the integrated analysis dataset; comparing the synchronization index data with preset synchronization requirements; if the synchronization index data meets the preset synchronization requirements, marking the analysis dataset as a qualified dataset; otherwise, correcting the analysis dataset; updating the synchronization index data and re-comparing it after correcting the analysis dataset; performing a final data integrity check on the qualified dataset and generating a verification result report; determining whether the analysis dataset can be used as the final output based on the verification result report; and storing the verified analysis dataset as the final gene screening result.
[0039] Specifically, to address the reliability issues of final screening results caused by asynchronous task execution and incomplete data in gene data analysis, the validation process first extracts synchronization index data from the integrated whole-genome analysis dataset. Synchronization is quantified by calculating the standard deviation of execution time for each data segment and comparing this value with preset synchronization requirements. If the standard deviation is less than a threshold, the dataset is directly marked as a qualified dataset; otherwise, a correction process is initiated. This involves identifying high-complexity segments, such as those with gene density greater than twice the average, and recalculating their load weight values using a dynamic programming algorithm (e.g., load weight = density × length / average). Then, a task migration protocol is used to transfer some data subtasks to nodes with loads below a preset threshold. After updating the execution time progress, the synchronization index data is recalculated and iteratively compared until the requirements are met. Subsequently, a data integrity check is performed on the qualified dataset to verify the completeness and completeness of key site information, generating a validation result report containing the number of sites and their missing information. Finally, based on the report, the dataset is confirmed as output and stored as the millet gene screening results.
[0040] By combining the above-mentioned synchronization verification and dynamic correction mechanisms, the risks of time deviation and data loss in the parallel analysis process are effectively eliminated, ensuring the consistency and accuracy of gene screening results.
[0041] Through the coordination of the above steps, this application achieves dynamic optimization and precise control of the entire millet gene data processing chain, significantly improving the resource utilization and analysis efficiency of large-scale parallel computing.
[0042] In summary, this application achieves systematic optimization of the millet gene screening process by constructing a complete technical chain from data preprocessing to result verification. Specifically, it establishes a precise load quantification system by employing an intelligent clustering algorithm based on density and length features, combined with a dynamic programming resource demand assessment mechanism; it forms a dynamic-static load balancing scheme through real-time node status monitoring and task migration driven by a greedy algorithm; it solves the distributed result consistency problem by utilizing a site association matrix and a majority voting mechanism; and it ensures output quality through synchronization verification and integrity checks. This scheme, through the synergistic effect of each stage, achieves significant improvements in data processing efficiency, resource utilization, and result accuracy, providing reliable technical support for large-scale genome analysis. The technical effects achievable by this application are shown in Figure 3, a radar chart of the performance indicators of the millet gene screening system. Example 2
[0043] The data mining-based millet gene screening method in this application embodiment has been described above. The data mining-based millet gene screening system in this application embodiment is described below. Referring to Figure 4, the data mining-based millet gene screening system in this application embodiment includes: a segmentation unit, used to acquire millet gene data and perform initial clustering processing on the millet gene data, dividing the millet gene data into multiple data segments to obtain a segment set; an indicator evaluation unit, used to evaluate the resource requirement indicators of each data segment in the segment set and determine the task allocation requirements; and a task distribution unit, used to acquire the current load status data of each computing node and distribute the data. Data segments are allocated to corresponding computing nodes, generating task distribution maps; a signal adjustment unit is used to start gene analysis tasks on computing nodes using a parallel execution mechanism and monitor the execution progress of each computing node. When the execution progress differs from a preset time threshold, a real-time synchronization adjustment signal is generated; a data integration unit is used to migrate and adjust tasks based on the real-time synchronization adjustment signal, optimize the task execution sequence, collect gene analysis results from each computing node from the optimized task execution sequence, and fuse key site information to obtain an integrated analysis dataset; a gene screening unit is used to verify the analysis dataset and output the final millet gene screening results.
[0044] Through the synergistic cooperation of the various components mentioned above, the resource utilization and analytical efficiency of large-scale parallel computing are further improved. Example 3
[0045] This application also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, storing instructions that, when executed on a computer, cause the computer to perform the steps of a millet gene screening method based on data mining.
[0046] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0047] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0048] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A data mining-based method for screening millet genes, characterized in that, The method includes: Step S1: Acquire millet gene data and perform initial clustering processing on the millet gene data to divide the millet gene data into multiple data segments, obtaining a segment set; Step S2: Evaluate the resource requirement index of each data segment in the segment set to determine the task allocation requirements; Step S3: Acquire the current load status data of each computing node, allocate the data segments to the corresponding computing nodes, and generate a task distribution mapping; Step S4: Start the gene analysis task on the computing node using a parallel execution mechanism and monitor the execution progress of each computing node. When the execution progress differs from a preset time threshold, a real-time synchronization adjustment signal is generated; Step S5: Based on the real-time synchronization adjustment signal, migrate and adjust the task, optimize the task execution sequence, collect the gene analysis results of each computing node from the optimized task execution sequence, fuse key site information, and obtain an integrated analysis dataset; Step S6: Perform validation processing on the analysis dataset and output the final millet gene screening results.
2. The method according to claim 1, characterized in that, Step S1 involves initial clustering of the millet gene data, including: Step S11: Processing the millet gene data using a preset clustering algorithm to divide it into multiple uniform data segments based on data density and length characteristics; Step S12: Preliminarily classifying the data segments to generate a segment set containing multiple data segments, extracting feature parameters for each data segment in the segment set, recording the correspondence between the feature parameters and segment identifiers, and verifying the segment set using the feature parameters; Step S13: Determining whether the segment set meets the uniformity condition. If it does, the segment set is used as input data for resource demand assessment; if it does not, the clustering parameters are adjusted and clustering is performed again; Step S14: Repeating step S13 until a segment set that meets the uniformity condition is obtained.
3. The method according to claim 1, characterized in that, Step S2 further includes: calculating the computational complexity of each data segment in the segment set using a dynamic programming algorithm; determining the load weight value of each data segment based on the computational complexity and the difference in millet gene density within the data segment; generating resource requirement indicators for the corresponding data segment based on the load weight value; sorting the resource requirement indicators to generate a resource requirement priority list; marking high-complexity data segments and low-complexity data segments according to the resource requirement priority list; comparing the resource requirement indicators of the high-complexity data segments with a preset indicator threshold to determine whether resources need to be allocated preferentially; marking the data segment as a high-priority task if the resource requirement indicator exceeds the preset indicator threshold; updating the allocation strategy of the resource requirement indicators based on the marking results, and using the updated resource requirement indicators as the basis for task allocation.
4. The method according to claim 3, characterized in that, Step S3, which involves allocating the data segments to the corresponding computing nodes, includes: obtaining the current load status data of each computing node and comparing the load status data of each computing node with a preset load threshold; if the load status data of a computing node is lower than the preset load threshold, then marking the computing node as an available node; according to the resource requirement index, prioritizing the allocation of the high-complexity data segments to the available nodes, and generating an initial task distribution map through the allocation process; performing load balancing verification on the task distribution map to determine whether the load of each computing node is balanced; if not, adjusting the data segment allocation scheme in the task distribution map until a task distribution map that meets the load balancing conditions is obtained, and using the task distribution map as the basic configuration for parallel task execution.
5. The method according to claim 1, characterized in that, Step S4 further includes: initializing gene analysis tasks on multiple computing nodes according to the task distribution mapping, starting the processing flow of the gene analysis tasks through a parallel execution mechanism; monitoring the execution time progress of the gene analysis tasks on each computing node in real time, comparing the execution time progress with the preset time threshold, determining whether there is a time difference, and if so, generating a real-time synchronization adjustment signal; otherwise, continuing to monitor the execution time progress of the gene analysis tasks.
6. The method according to claim 1, characterized in that, Step S5, which involves migrating and adjusting tasks, includes: receiving and parsing the task segments marked in the real-time synchronization adjustment signal that need adjustment, identifying the current computing node and execution status of the task segment; determining the remaining computing load of the task segment based on the execution status, finding the computing node with a lower load based on the remaining computing load, and transferring some of the computing tasks of the task segment to the computing node with a lower load using a task migration protocol; updating the task distribution mapping through task transfer, and predicting the execution time of the updated task distribution mapping to determine whether the overall execution time is balanced. If it is not balanced, the task allocation is further adjusted. After multiple adjustments, an optimized task execution sequence is generated and used as the final task scheduling scheme.
7. The method according to claim 6, characterized in that, Step S5, which involves collecting gene analysis results from each computing node in the optimization task execution sequence, includes: obtaining the gene analysis task results completed by each computing node according to the optimization task execution sequence; extracting key site information from the gene analysis task results; performing data cleaning and format unification processing using the key site information; fusing the processed key site information to generate a preliminary integrated dataset; performing consistency verification on the preliminary integrated dataset to determine if there are any data conflicts; if so, correcting the conflicting data; if not, determining the preliminary integrated dataset as the integrated analysis dataset.
8. The method according to claim 1, characterized in that, Step S6 further includes: acquiring and extracting synchronization index data from the integrated analysis dataset; comparing the synchronization index data with preset synchronization requirements; if the synchronization index data meets the preset synchronization requirements, marking the analysis dataset as a qualified dataset; otherwise, correcting the analysis dataset; updating the synchronization index data and re-comparing it after correcting the analysis dataset; performing a final data integrity check on the qualified dataset and generating a verification result report; determining whether the analysis dataset can be used as the final output based on the verification result report; and storing the verified analysis dataset as the final gene screening result.
9. A data mining-based millet gene screening system, used to implement the data mining-based millet gene screening method as described in any one of claims 1-8, characterized in that, The system includes: a segmentation unit for acquiring millet gene data and performing initial clustering processing on the millet gene data to divide the millet gene data into multiple data segments, obtaining a segment set; an indicator evaluation unit for evaluating the resource requirement indicators of each data segment in the segment set and determining task allocation requirements; a task distribution unit for acquiring the current load status data of each computing node, allocating the data segments to the corresponding computing nodes, and generating a task distribution mapping; a signal adjustment unit for starting gene analysis tasks on the computing nodes using a parallel execution mechanism and monitoring the execution progress of each computing node, generating a real-time synchronization adjustment signal when the execution progress differs from a preset time threshold; a data integration unit for migrating and adjusting tasks according to the real-time synchronization adjustment signal, optimizing the task execution sequence, collecting gene analysis results from each computing node from the optimized task execution sequence, fusing key site information, and obtaining an integrated analysis dataset; and a gene screening unit for verifying the analysis dataset and outputting the final millet gene screening results.
10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the millet gene screening method based on data mining as described in any one of claims 1-8.