Resource allocation method and device, computer equipment and computer storage medium

Abstract

The invention relates to a resource configuration method and device, computer equipment and a computer storage medium, and the method comprises the steps: firstly generating multiple batches of raw information processing tasks based on pre-obtained biological information original data; performing configuration analysis processing on the first batch of the raw signal processing tasks to obtain target resource configuration information of the raw signal processing tasks; and according to the target resource configuration information, performing resource configuration processing on the remaining batches of the raw signal processing tasks to obtain resource configuration results corresponding to the raw signal processing tasks. According to the method, the target resource configuration information is determined, various types of data collected in the first batch of task processing process can be fully utilized, and resource configuration processing is carried out on the remaining batches of signal processing tasks according to the target resource configuration information, so that the data processing process can be carried out under a unified and optimized resource configuration framework; and it is ensured that all the raw signal processing tasks can be efficiently executed in a proper resource environment.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a resource configuration method, apparatus, computer equipment, and computer storage medium. Background Art

[0002] As an interdisciplinary field, bioinformatics integrates knowledge from multiple disciplines such as biology, computer science, mathematics and statistics, and plays a vital role in modern life science research. With the development of high-throughput sequencing and analysis technology and the explosive growth of biological data, bioinformatics provides key support for understanding complex biological systems and revealing the mysteries of life.

[0003] Limited by the computing power bottleneck of personal computers, analyzing massive biological data through computing platforms on public clouds has become an important research method for scientists in the field. However, the current computing platforms have low efficiency in allocating computing resources when processing these biological data. Summary of the invention

[0004] Based on this, it is necessary to provide a resource configuration method, device, computer equipment and computer storage medium that can solve the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a resource configuration method, the method comprising:

[0006] Generate multiple batches of bioinformatics processing tasks based on pre-acquired bioinformatics raw data;

[0007] Perform configuration analysis on the first batch of bioinformatics processing tasks to obtain target resource configuration information for the bioinformatics processing tasks;

[0008] The remaining batches of bioinformatics processing tasks are resource configured according to the target resource configuration information to obtain the resource configuration results corresponding to each bioinformatics processing task.

[0009] In one embodiment, the configuration analysis and processing of the first batch of bioinformatics processing tasks is performed to obtain target resource configuration information of the bioinformatics processing tasks, including:

[0010] Determine the job distribution resources corresponding to the first batch of bioinformatics processing tasks based on the pre-established basic experience strategy set and the pre-acquired bioinformatics job flow;

[0011] Execute the first batch of bioinformatics processing tasks according to the job distribution resources, and obtain the task processing data of each sample in the first batch of bioinformatics processing tasks;

[0012] Determine the target resource configuration information based on the task processing data of each sample in the first batch of bioinformatics processing tasks.

[0013] In one embodiment, the above-mentioned determining the target resource configuration information according to the task processing data of each sample in the first batch of bioinformatics processing tasks includes:

[0014] According to the task processing data of each sample in the first batch of bioinformatics processing tasks, multiple multi-process single-Megabyte data processing times are determined; the number of processes corresponding to each multi-process single-Megabyte data processing time is different;

[0015] Normalize the single-Megabyte data processing time of each multi-process respectively to obtain multiple single-process single-Megabyte data processing times;

[0016] The resource configuration information with the shortest processing time for single-process single-M data is determined as the target resource configuration information.

[0017] In one embodiment, the resource configuration processing is performed on the remaining batches of bioinformatics processing tasks according to the target resource configuration information to obtain the resource configuration results corresponding to each bioinformatics processing task, including:

[0018] Determine the resource requirements for the remaining batches of bioinformatics processing tasks based on the target resource allocation information;

[0019] Determine multiple candidate configuration solutions based on resource requirements and multiple pre-acquired computing instances;

[0020] Determine the target configuration scheme based on each candidate configuration scheme and the pre-built optimal solution model;

[0021] According to the target configuration plan, resource configuration is performed on the remaining batches of bioinformatics processing tasks to obtain resource configuration results.

[0022] In one embodiment, the above-mentioned determining the target configuration scheme according to each candidate configuration scheme and the pre-built optimal solution model includes:

[0023] Based on the optimal solution model, the performance indicator prediction processing is performed on each candidate configuration scheme to obtain the performance indicator prediction value corresponding to each candidate configuration scheme;

[0024] Determine the comprehensive performance score corresponding to each candidate configuration scheme according to the pre-acquired performance indicator weights and the predicted values ​​of each performance indicator;

[0025] The candidate configuration scheme with the highest comprehensive performance score is determined as the target configuration scheme.

[0026] In one embodiment, the method comprises:

[0027] When the resource demand of the target configuration solution is greater than the available resources of the current platform, the cloud computing platform is expanded;

[0028] When the resource load value of a single computing instance indicates no load and the bioinformatics processing task corresponding to the single computing instance indicates task completion, the cloud computing platform is scaled down.

[0029] In one embodiment, the above-mentioned expansion processing of the cloud computing platform includes:

[0030] Determine the computing instance to be added based on the resource demand;

[0031] The computing instance interface creation process is performed on the cloud computing platform according to the computing instance to be added.

[0032] In a second aspect, the present application also provides a resource configuration device. The device includes:

[0033] A task generation module is used to generate multiple batches of bioinformatics processing tasks based on pre-acquired bioinformatics raw data;

[0034] The analysis module is used to perform configuration analysis on the first batch of bioinformatics processing tasks to obtain the target resource configuration information of the bioinformatics processing tasks;

[0035] The configuration module is used to perform resource configuration processing on the remaining batches of bioinformatics processing tasks according to the target resource configuration information to obtain the resource configuration results corresponding to each bioinformatics processing task.

[0036] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0037] Generate multiple batches of bioinformatics processing tasks based on pre-acquired bioinformatics raw data;

[0038] Perform configuration analysis on the first batch of bioinformatics processing tasks to obtain target resource configuration information for the bioinformatics processing tasks;

[0039] The remaining batches of bioinformatics processing tasks are resource configured according to the target resource configuration information to obtain the resource configuration results corresponding to each bioinformatics processing task.

[0040] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0041] Generate multiple batches of bioinformatics processing tasks based on pre-acquired bioinformatics raw data;

[0042] Perform configuration analysis on the first batch of bioinformatics processing tasks to obtain target resource configuration information for the bioinformatics processing tasks;

[0043] The remaining batches of bioinformatics processing tasks are resource configured according to the target resource configuration information to obtain the resource configuration results corresponding to each bioinformatics processing task.

[0044] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0045] Generate multiple batches of bioinformatics processing tasks based on pre-acquired bioinformatics raw data;

[0046] Perform configuration analysis on the first batch of bioinformatics processing tasks to obtain target resource configuration information for the bioinformatics processing tasks;

[0047] The remaining batches of bioinformatics processing tasks are resource configured according to the target resource configuration information to obtain the resource configuration results corresponding to each bioinformatics processing task.

[0048] The above-mentioned resource configuration method, device, computer equipment and computer storage medium first generate multiple batches of bioinformatics processing tasks based on the pre-acquired raw bioinformatics data; perform configuration analysis on the first batch of bioinformatics processing tasks to obtain the target resource configuration information of the bioinformatics processing tasks; perform resource configuration processing on the remaining batches of bioinformatics processing tasks according to the target resource configuration information to obtain the resource configuration results corresponding to each bioinformatics processing task. This application determines the target resource configuration information, which can make full use of various types of data collected during the processing of the first batch of tasks, and performs resource configuration processing on the remaining batches of bioinformatics processing tasks according to the target resource configuration information, so that the data processing process can be carried out under a unified and optimized resource configuration framework, ensuring that the bioinformatics processing tasks can be efficiently executed in a suitable resource environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 An application environment diagram of a resource configuration method in an embodiment;

[0050] Figure 2 A schematic diagram of a resource configuration method in one embodiment;

[0051] Figure 3 A schematic diagram of a process for obtaining target resource configuration information of a bioinformatics processing task in one embodiment;

[0052] Figure 4 A schematic diagram of a process for determining target resource configuration information in an embodiment;

[0053] Figure 5 A schematic diagram of a process for obtaining resource configuration results corresponding to each bioinformatics processing task in an embodiment;

[0054] Figure 6A schematic diagram of a process for determining a target configuration scheme in an embodiment;

[0055] Figure 7 A schematic diagram of a flow chart of a capacity expansion process in an embodiment;

[0056] Figure 8 A structural block diagram of a resource configuration device in an embodiment;

[0057] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] First of all, before specifically introducing the technical solution of the embodiments of the present application, the technical background based on the embodiments of the present application is introduced.

[0060] The importance and application scope of bioinformatics are expanding. With the development of high-throughput sequencing and analysis technologies and the explosive growth of biological data, bioinformatics provides key support for understanding complex biological systems and revealing the mysteries of life. Limited by the computing power bottleneck of personal computers, analyzing massive biological data through computing platforms on public clouds has become an important research method for scientists in the field. Bioinformatics has not only made rapid progress in basic research such as genomics, transcriptomics, proteomics and metabolomics, but has also brought changes to application fields such as pathological research, drug development and agricultural improvement.

[0061] With the rapid development of biology, the processing flow in its field has gradually become more refined, such as cell biology, genetics, evolutionary biology, etc., and the data generated and the required analysis methods have become more diverse; the development of computational science is also changing with each passing day, and a wide variety of computing platforms provide rich choices for different demand scenarios. The diversity of needs and the complexity of tools in the field of bioinformatics urgently require the sorting and classification of business needs in the field and the convergence of the flexibility of computing tools to achieve efficient, stable and cost-effective implementation of bioinformatics data analysis. The current bioinformatics analysis platform usually cannot take into account the effective support of industry software and the high-performance release of computing power. The business software is industry-oriented and despises the implementation of the computing power platform. The existing implementation on the computing power platform focuses on single-point performance optimization and ignores the business characteristics of bioinformatics analysis. In view of the interdisciplinary nature of bioinformatics, it is necessary to provide a public cloud adaptive resource supply method for bioinformatics to achieve efficient calculation and analysis of bioinformatics data.

[0062] Based on this, the present application provides a resource configuration method, apparatus, computer device and computer storage medium, aiming to solve the above technical problems.

[0063] The resource configuration method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the public cloud 104 through the network. The data storage system can store the data that the public cloud 104 needs to process. The data storage system can be integrated on the public cloud 104, or it can be placed on the cloud or other network public clouds. The public cloud generates multiple batches of bioinformatics processing tasks based on the original bioinformatics data pre-acquired by 104; performs configuration analysis and processing on the first batch of bioinformatics processing tasks to obtain the target resource configuration information of the bioinformatics processing tasks; performs resource configuration processing on the remaining batches of bioinformatics processing tasks according to the target resource configuration information to obtain the resource configuration results corresponding to each bioinformatics processing task. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones and tablets. The public cloud 104 can be implemented with an independent public cloud or a public cloud cluster composed of multiple public clouds.

[0064] In one embodiment, Figure 2 As shown, a resource configuration method is provided, which is applied to Figure 1 The public cloud platform in is used as an example to illustrate the process, including the following steps:

[0065] S201, generating multiple batches of bioinformatics processing tasks based on pre-acquired bioinformatics raw data.

[0066] Among them, raw bioinformatics data refers to the unprocessed raw bioinformatics data directly obtained in the process of biological science research, such as gene sequence data obtained by gene sequencing instruments, microbial community composition data collected from ecological environment samples, etc. These data contain rich biological information, but usually need to go through a series of complex bioinformatics processing steps to be transformed into valuable research results.

[0067] Bioinformatics processing tasks are a series of operational tasks based on raw bioinformatics data to extract, analyze and interpret the bioinformatics contained therein. It covers multiple processing links, such as data quality control (removing low-quality data fragments), gene alignment (aligning the sequenced gene sequence with the known reference genome), functional analysis (determining the function of the gene and its role in the organism, etc.), etc. Each link has its specific computing resource requirements.

[0068] In an embodiment of the present application, first, the acquired raw bioinformatics data is preprocessed, such as checking the integrity of the data, format conversion, and other operations, to ensure that the data can be correctly read and processed by subsequent processing procedures. Then, the raw data is divided into multiple batches according to the size of the data, the complexity of the processing, and the current resource status of the public cloud platform. For example, if the amount of raw data is large and the processing task is more complex, and the resources of the public cloud platform are limited, it can be divided into smaller batches to reasonably allocate resources and avoid system overload. For each batch, the bioinformatics processing tasks that need to be performed in the batch are determined according to the preset bioinformatics processing flow template, including the specific operations and sequence of each link.

[0069] Another implementation method: When dividing batches, a division method based on data features can be used. First, extract features from the original bioinformatics data, such as analyzing the length distribution of gene sequence data and the species diversity of microbial community data. Then divide the data into different subsets based on these features, and each subset is a batch. This method can make the data of each batch more similar in processing characteristics, which is convenient for subsequent targeted resource allocation based on batch characteristics. At the same time, when determining the bioinformatics processing tasks for each batch, the complexity and order of the tasks can be dynamically adjusted according to the load of the public cloud platform resources monitored in real time. For example, if the computing core resources of the current platform are relatively tight, but the memory resources are relatively sufficient, the proportion of tasks with higher computing core requirements in the batch can be appropriately reduced, and tasks with greater memory dependence can be increased.

[0070] S202, performing configuration analysis on the first batch of bioinformatics processing tasks to obtain target resource configuration information of the bioinformatics processing tasks.

[0071] The target resource configuration information is a set of optimal resource configuration solutions determined after a detailed analysis of the first batch of bioinformatics processing tasks, including but not limited to the number of computing cores required for each processing link, memory allocation, optimal thread configuration, etc. It can ensure that bioinformatics processing tasks can maximize resource utilization efficiency while meeting processing requirements.

[0072] The embodiment of the present application submits the first batch of bioinformatics processing tasks to the public cloud platform for processing, and starts the runtime monitoring module to comprehensively monitor the processing process of the task. The monitoring content includes but is not limited to indicators such as the processing time of each task in each processing link, the memory usage under different thread configurations, and the computing core utilization rate. By collecting these data, the performance of the first batch of bioinformatics processing tasks under different resource configurations is analyzed in detail. For example, for the gene comparison link, the processing time and memory usage under different thread configurations (such as 8 threads, 16 threads, etc.) are compared to find out the thread configuration method that can minimize the processing time and maximize resource utilization while ensuring the processing quality. Based on the analysis results of each processing link, the target resource configuration information of the bioinformatics processing task is determined, such as the optimal thread configuration for each link, the required number of computing cores, and the memory allocation scheme.

[0073] Another implementation method: In addition to the data collected by the runtime monitoring module, historical task data can also be introduced as a reference. That is, query the relevant data of similar bioinformatics tasks that have been processed in the past on the public cloud platform, including their resource configuration and processing results. These historical data are comprehensively analyzed with the data collected during the processing of the first batch of bioinformatics processing tasks. For example, if a gene analysis task in the historical tasks has a better processing effect under a specific thread configuration, and the same link in the first batch of tasks also shows certain advantages under this thread configuration, then the applicability of the thread configuration can be more determined. In this way, combining real-time monitoring data and historical experience data, the target resource configuration information of the bioinformatics processing task can be determined more comprehensively and accurately.

[0074] S203: Perform resource configuration processing on the remaining batches of bioinformatics processing tasks according to the target resource configuration information to obtain resource configuration results corresponding to each bioinformatics processing task.

[0075] The embodiment of the present application calculates the total amount of resources required for the remaining batches of bioinformatics processing tasks according to the target resource configuration information determined in step S202. For example, if the target resource configuration information determines that each sample in the gene comparison link requires 4 computing cores and 8GB of memory, and there are 100 samples in the remaining batches, then the remaining batches of tasks in this link require a total of 400 computing cores and 800GB of memory. Submit a resource application request to the public cloud platform to clarify the required computing instance type (determined according to the computing core and memory requirements), quantity, and memory allocation and other detailed information. The public cloud platform allocates and configures resources according to its own resource conditions, and prepares the corresponding computing environment for the remaining batches of bioinformatics processing tasks. Then, the remaining batches of bioinformatics processing tasks are allocated according to the target resource configuration information, such as configuring the corresponding number of threads for each task, arranging them to run on the appropriate computing instance, etc., and finally obtaining the resource configuration results corresponding to each bioinformatics processing task.

[0076] Another implementation method: When performing resource configuration processing according to the target resource configuration information, a dynamic adjustment strategy can be adopted. During the execution of the remaining batches of bioinformatics processing tasks, continue to use the runtime monitoring module to monitor the processing of the tasks in real time. If it is found that the current target resource configuration information is no longer the optimal choice due to changes in data characteristics or fluctuations in platform resource conditions, for example, the processing time of a task begins to increase or the resource utilization rate becomes unreasonable, the target resource configuration information is adjusted in a timely manner. The amount of resources required for the task can be recalculated, and the thread configuration, computing instance allocation, etc. can be changed to adapt to the new situation, ensuring that the remaining batches of bioinformatics processing tasks are always carried out under the optimal resource configuration, and obtaining resource configuration results that are more in line with the actual situation.

[0077] The above-mentioned resource configuration method, device, computer equipment and computer storage medium first generate multiple batches of bioinformatics processing tasks based on the pre-acquired raw bioinformatics data; perform configuration analysis on the first batch of bioinformatics processing tasks to obtain the target resource configuration information of the bioinformatics processing tasks; perform resource configuration processing on the remaining batches of bioinformatics processing tasks according to the target resource configuration information to obtain the resource configuration results corresponding to each bioinformatics processing task. This application determines the target resource configuration information, which can make full use of various types of data collected during the processing of the first batch of tasks, and performs resource configuration processing on the remaining batches of bioinformatics processing tasks according to the target resource configuration information, so that the data processing process can be carried out under a unified and optimized resource configuration framework, ensuring that the bioinformatics processing tasks can be efficiently executed in a suitable resource environment.

[0078] In an exemplary embodiment, based on the above embodiment, see Figure 3 The embodiment of the present application relates to a process of performing configuration analysis on a first batch of bioinformatics processing tasks to obtain target resource configuration information of the bioinformatics processing tasks, including the following steps:

[0079] S301, based on a pre-established basic experience strategy set and a pre-acquired bioinformatics operation process, determine the job distribution resources corresponding to the first batch of bioinformatics processing tasks.

[0080] Among them, the basic empirical strategy set is a set of experience summaries of resource allocation for different types of bioinformatics processing tasks (such as gene sequencing analysis, protein structure prediction, etc.) under common data scales and processing scenarios, which has been accumulated in the long-term practice of bioinformatics data processing. It covers the approximate resource ranges usually required for different task links (such as data acquisition, quality control, analytical calculations, etc.), including but not limited to empirical suggestions on the number of computing cores, memory allocation, thread configuration, etc., as a reference for the preliminary determination of the first batch of bioinformatics processing task resource allocation.

[0081] The bioinformatics operation process is the business process of bioinformatics. Most of them start with obtaining raw data, and then perform quality control on the raw data (i.e., quality control process, hereinafter referred to as quality control). The quality control process will evaluate the proportion of genes in the raw data entries, and use this as a basis to screen raw data entries with good quality; at the same time, the quality control process will also remove duplicates and noise from the raw data that pass the quality control (deduplication refers to repeated reads obtained in the raw data due to sampling, and denoising refers to removing reads of other non-target organisms in the environment). After the quality control process is completed, this batch of data will be disassembled and merged. This process merges scattered and repeated reads into biologically meaningful genes, thereby generating a set of gene data in the sampled sample. After obtaining the genes in the sample, genetic analysis is carried out by comparing and mapping with various reference genomes (such as gene family reference genomes, protein-coding gene reference genomes, evolutionary reference genomes, etc.), including gene component analysis, family analysis, process analysis and other analysis processes. Through these analyses (hereinafter referred to as analysis), the gene family, functional (protein) cause, evolutionary process, etc. of the sampling subject can be further determined. The output results of these analyses include information such as species abundance, gene detection coverage, gene family genealogy of the analyzed subject, and the evolutionary process of the analyzed subject, which are essential data for biological scientists in their research work.

[0082] This application performs a performance analysis of each step of the bioinformatics workflow. For the data acquisition stage, because the network bandwidth is often smaller than the bandwidth of the data being dropped to disk, when the total network bandwidth is constant, the time to acquire the original data is linearly related to the size of the data itself; for the quality control process, subject to objective factors such as the sampling method, sequencing method, and environmental noise, the time required for processing is random, which is reflected in the fact that the processing time of the original data of the same size of storage space may have a large deviation; the time required for the disassembly and merging process is relatively short, and the time required is linearly related to the size of the data; the gene analysis process is to compare with the reference genome, and the time consumption of this process is linearly related to the size of the gene file to be processed. Because there is no mutual influence between genes, the gene comparison process can be parallel between gene data at the algorithm implementation level. We can assign a group of processes to a gene for comparison, and there is no need for communication between the processes used by different genes.

[0083] We conducted multiple sample tests on the gene analysis process and built a basic empirical strategy set. After performance testing, the basic experience is that the use of hyperthreading will bring performance benefits. At the same time, the memory demand of a single task is related to the size of the reference genome and the size of the genes to be processed. The empirical strategy set recommends several computing resources (i.e., computing instances) to ensure that the task can run.

[0084] This application aims to achieve the adaptability of bioinformatics sample processing and resource supply. In view of the fact that the data acquisition part is mainly affected by the network bandwidth, and the time consumption of the disassembly and merging process accounts for a very small proportion of the overall processing time, this application chooses not to model and adaptively optimize the time consumption of these two links. On the contrary, this application focuses on the two key links of quality control and gene analysis for adaptive optimization. The quality control part has a high sporadic nature of genome sequence alignment events, and the processing time is highly random; while the gene analysis part needs to align all genome sequences, and its processing time is relatively fixed and predictable. Based on this feature, this application regards the optimal resource supply problem as a two-dimensional packing optimization problem, and constructs a performance optimal solution model for genetic data, process requirements, and memory requirements. Through this model, the number of processes and memory capacity required for a single bioinformatics task can be optimized to achieve the maximum throughput efficiency for the overall sample to be analyzed in the cloud computing platform. In terms of memory usage, the memory size required for a single bioinformatics raw data should comprehensively consider the impact of the two processes of quality control and analysis. For the number of processes required for a single sample, it is necessary to consider the optimal solution to achieve performance in the analysis phase to ensure the processing efficiency of a single sample. In this way, resource allocation can be effectively optimized and the overall processing capability of bioinformatics tasks can be improved.

[0085] In the embodiment of the present application, the pre-acquired bioinformatics operation process is first analyzed in detail to clarify the specific task content, data input and output requirements, and expected processing complexity of each processing link. For example, for the bioinformatics operation process of gene analysis, it is necessary to determine the data source and data volume range of the gene sequencing data acquisition link, the quality standard and processing algorithm complexity of the quality control link, the comparison algorithm of the gene comparison link, and the required reference genome and other information. Then, according to the analysis results, the task type and processing scenario that match or are similar to it are found in the pre-established basic experience strategy set. Taking gene analysis as an example, if the data scale and processing complexity of the current task are similar to a medium-sized gene sequencing analysis scenario in the basic experience strategy set, then refer to the resource allocation experience in this scenario. Then, combined with the actual characteristics of this bioinformatics operation process, the empirical resource allocation suggestions found are appropriately adjusted. For example, if the gene analysis task needs to obtain data from multiple different data sources in the data acquisition link, it may require slightly more computing cores than similar scenarios in the basic experience strategy to accelerate data transmission and integration, thereby determining the job distribution resources corresponding to the first batch of bioinformatics processing tasks, including the specific number of computing cores, memory allocation, thread configuration, etc.

[0086] S302, executing the first batch of bioinformatics processing tasks according to the job distribution resources, and obtaining the task processing data of each sample in the first batch of bioinformatics processing tasks.

[0087] In the embodiment of the present application, according to the job distribution resources determined in step S301, the first batch of bioinformatics processing tasks are assigned to the corresponding computing resources of the public cloud platform or the local computing environment (depending on the actual situation) for execution. During the execution process, the runtime monitoring module is started to comprehensively monitor the processing of each sample in each processing link. The monitoring content includes but is not limited to: the start time and end time of each sample in different processing links, so as to calculate the processing time; the utilization rate of the computing core in each link to determine the computing core utilization rate; the occupancy and change of the memory in each processing link, that is, the memory occupancy; and the change of the quality index of the data during the processing process (such as the improvement of the accuracy of the sequencing data after the quality control link, etc.). By collecting and sorting these data, the task processing data of each sample in the first batch of bioinformatics processing tasks are obtained. For example, for a sample in a gene analysis task, it is recorded that it took 5 minutes in the gene sequencing data acquisition link, the computing core utilization rate was 30%, the memory occupancy was 2GB, and the accuracy of the sequencing data after the quality control link was improved from 80% to 90%. Detailed information such as.

[0088] S303: Determine target resource configuration information according to the task processing data of each sample in the first batch of bioinformatics processing tasks.

[0089] First, the embodiment of the present application classifies and analyzes the task processing data of each sample in the first batch of bioinformatics processing tasks. For example, the processing time, computing core utilization, memory usage and other data of different samples in the same processing link (such as gene comparison link) are summarized and averaged to obtain the overall performance data of the link under the current job distribution resource configuration. Then, for each processing link, by comparing the average processing time, resource utilization and other data of different samples, find a resource configuration method that can minimize the processing time and maximize resource utilization while ensuring the processing quality. Taking the gene comparison link as an example, if it is found that when the thread configuration is 16, the average processing time is the shortest and the computing core utilization and memory usage are both in a more reasonable range, then the optimal thread configuration of the gene comparison link is preliminarily determined to be 16. Next, the analysis results of each processing link are combined to determine the target resource configuration information of the bioinformatics processing task, including the optimal thread configuration of each link, the required number of computing cores, the memory allocation scheme, etc. For example, after analyzing each link of the gene analysis task, it was determined that 2 computing cores and 4GB of memory are required in the data acquisition link, 3 computing cores and 6GB of memory are required in the quality control link, and the optimal thread configuration in the gene comparison link is 16 and requires 4 computing cores and 8GB of memory, etc., and complete target resource configuration information.

[0090] The embodiment of the present application preliminarily determines the job distribution resources based on the basic experience strategy set and the bioinformatics job process, and can provide a relatively reasonable resource allocation reference when the first batch of bioinformatics processing tasks are started, avoiding resource waste or task processing difficulties caused by blind allocation of resources. This lays a good foundation for further optimizing resource allocation through detailed analysis of the processing data of the first batch of tasks, so that the final target resource allocation information is more accurately in line with the actual needs of bioinformatics data processing tasks.

[0091] In an exemplary embodiment, based on the above embodiment, see Figure 4 , the embodiment of the present application relates to a process of determining target resource configuration information according to task processing data of each sample in the first batch of bioinformatics processing tasks, including the following process:

[0092] S401, determining a plurality of multi-process single-M data processing times according to the task processing data of each sample in the first batch of bioinformatics processing tasks, wherein the number of processes corresponding to each multi-process single-M data processing time is different.

[0093] The multi-process single-megabyte data processing time refers to the time it takes to process one megabyte (1MB) of data for each sample under different process number configurations in the first batch of bioinformatics processing tasks. For example, when the number of processes is set to 4, the time it takes to process 1MB of data is one type of multi-process single-megabyte data processing time; when the number of processes becomes 8, the corresponding processing time is another type of multi-process single-megabyte data processing time. It reflects the time required to process a unit of data volume under different degrees of parallel processing, and is an important indicator for analyzing the impact of different resource configurations on processing efficiency.

[0094] The single-process single-megabyte data processing time is obtained by normalizing the multi-process single-megabyte data processing time, that is, converting the multi-process single-megabyte data processing time into the time required to process each megabyte of data based on a specific calculation method. In this way, the efficiency of processing unit data under different process number configurations can be compared under the same standard, making it easier to determine the optimal resource allocation plan.

[0095] In the embodiment of the present application, first, data information related to processing time and number of processes is extracted from the task processing data of each sample in the first batch of bioinformatics processing tasks. These data information include the processing start time, processing end time and the amount of data processed for each sample under different processing links (such as data acquisition, quality control, gene analysis, etc.) and different process number configurations. For example, for a gene analysis sample, the record shows that when the number of processes is 4, the gene comparison link starts processing at 9 am and ends at 10 am, and a total of 20MB of data is processed; when the number of processes is adjusted to 8, the processing starts at 10:10 am and ends at 11:20 am, processing 30MB of data. Then, based on the extracted data, for each processing link of each sample, the multi-process single-megabyte data processing time is calculated according to the following formula:

[0096] Multi-process single-megabyte data processing time = (processing end time - processing start time) / processed data volume (MB).

[0097] For example, for the gene comparison with 4 processes, the multi-process single-Megabyte data processing time = (10-9) hours / 20MB = 0.05 hours / MB; for the case with 8 processes, the multi-process single-Megabyte data processing time = (11.20-10.10) hours / 30MB = 0.0367 hours / MB. By performing such calculations on each processing link of each sample in the first batch of bioinformatics processing tasks, multiple multi-process single-Megabyte data processing times are determined, and each multi-process single-Megabyte data processing time corresponds to a different number of processes.

[0098] S402, normalizing the single-Megabyte data processing time of each multi-process to obtain multiple single-process single-Megabyte data processing times.

[0099] In the embodiment of the present application, the following normalized processing formula is used for processing each multi-process single-Megabyte data processing time determined in step S401: single-process single-Megabyte data processing time = multi-process single-Megabyte data processing time × number of processes.

[0100] For example, for the previously calculated multi-process single-Megabyte data processing time of 0.05 hours / MB with 4 processes, after normalization, the single-process single-Megabyte data processing time = 0.05 hours / MB × 4 = 0.2 hours / MB; for the multi-process single-Megabyte data processing time of 0.0367 hours / MB with 8 processes, after normalization, the single-process single-Megabyte data processing time = 0.0367 hours / MB × 8 = 0.2936 hours / MB. By performing such normalization on each multi-process single-Megabyte data processing time and converting it into a single-process single-Megabyte data processing time, the efficiency of processing unit data under different process number configurations can be compared under the same standard.

[0101] S403, determining the resource configuration information with the shortest single-process single-M data processing time as the target resource configuration information.

[0102] After obtaining multiple single-process single-megabyte data processing times, the embodiment of the present application compares these times. First, each single-process single-megabyte data processing time is associated with its corresponding resource configuration information (such as the number of processes, thread configuration, number of computing cores, memory allocation scheme, etc.). For example, the resource configuration information corresponding to the single-process single-megabyte data processing time of 0.2 hours / MB may be that the number of processes is 4, the thread configuration is 8, the number of computing cores is 2, and the memory allocation scheme is 4GB; the resource configuration information corresponding to another single-process single-megabyte data processing time of 0.2642 hours / MB may be that the number of processes is 8, the thread configuration is 16, the number of computing cores is 4, and the memory allocation scheme is 8GB. Then, among all the single-process single-megabyte data processing times, find the minimum time value. Assume that after comparison, it is found that 0.2 hours / MB is the minimum single-process single-megabyte data processing time. Finally, the resource configuration information corresponding to this minimum single-process single-megabyte data processing time is determined as the target resource configuration information. That is, the number of processes is determined to be 4, the thread configuration is 8, the number of computing cores is determined to be 2, and the memory allocation scheme is determined to be 4 GB as the target resource configuration information, which is used for resource configuration of subsequent batches of bioinformatics processing tasks.

[0103] The embodiment of the present application calculates the processing time of multiple multi-process single-Megabyte data and performs normalization processing, and finally determines the resource configuration information with the smallest single-process single-Megabyte data processing time as the target resource configuration information, which can accurately find the resource configuration scheme with the highest efficiency when processing unit data. This in-depth analysis method based on data processing time avoids the drawbacks of traditional resource configuration methods that determine resource configuration based on experience or rough estimates, making resource configuration more accurately fit the actual needs of bioinformatics data processing tasks, and improving the scientificity and accuracy of resource configuration.

[0104] In an exemplary embodiment, based on the above embodiment, see Figure 5 The embodiment of the present application relates to a process of performing resource configuration processing on the remaining batches of bioinformatics processing tasks according to target resource configuration information to obtain resource configuration results corresponding to each bioinformatics processing task, including the following steps:

[0105] S501: Determine the resource requirements of the remaining batches of bioinformatics processing tasks according to the target resource configuration information.

[0106] The embodiment of the present application first analyzes the target resource configuration information in detail to clarify the specific resource configuration requirements such as the number of computing cores and memory allocation required for each processing link (such as data acquisition, quality control, gene analysis, etc.). Then, combined with the actual situation of the remaining batches of bioinformatics processing tasks, the number of remaining samples and the amount of data expected to be processed by each sample in each processing link are mainly considered. For example, if the target resource configuration information shows that each sample in the gene analysis link requires 2 computing cores and 4GB of memory, and there are 50 samples in the remaining batch, then in the gene analysis link, the number of computing cores required for the remaining batch of bioinformatics processing tasks is 2×50=100, and the total memory is 4×50=200GB. In the same way, each processing link is calculated separately, and the amount of resources required for each link is summarized to determine the resource requirements for completing the remaining batches of bioinformatics processing tasks, including the total number of computing cores, the total amount of memory, and other resources that may be involved (such as storage capacity, etc., if applicable).

[0107] S502: Determine multiple candidate configuration solutions based on resource requirements and multiple pre-acquired computing instances.

[0108] Among them, computing instances refer to virtual computing units with specific computing resource configurations provided by cloud computing platforms, usually including a certain number of computing cores, memory capacity, and related network configurations, etc. Different computing instances have different resource specifications, which can be selected and combined according to task requirements to meet the resource requirements of bioinformatics data processing tasks.

[0109] Candidate configuration schemes are multiple possible resource configuration schemes formed by different combinations and resource allocation strategies based on resource requirements and multiple pre-acquired computing instances. Each candidate configuration scheme represents a way to match computing instances with the remaining batch of bioinformatics processing tasks, aiming to explore different resource utilization paths while meeting resource requirements, so as to select the optimal target configuration scheme.

[0110] The embodiment of the present application first obtains information on multiple computing instances of different specifications provided by the cloud computing platform. These computing instances have different numbers of computing cores, memory capacities, and possible other resource configuration features (such as storage capacity, network bandwidth, etc.). Then, based on the determined resource requirements, candidate configuration schemes are generated through different combinations and resource allocation strategies. For example, if the resource requirements require 200 computing cores and 800GB of memory, the cloud computing platform provides three computing instances: instance A (40 cores, 160GB of memory), instance B (20 cores, 80GB of memory), and instance C (10 cores, 40GB of memory). One candidate configuration scheme may be to use 5 instances of A, i.e., 5×40=200 computing cores, 5×160=800GB of memory; another candidate configuration scheme may be to use 10 instances of B and 2 instances of C, i.e., (10×20+2×10)=200 computing cores, (10×80+2×40)=880GB of memory. In this way, multiple candidate configuration schemes are generated according to different computing instance combinations and resource allocation strategies to meet resource requirements and explore different resource utilization paths.

[0111] S503: Determine a target configuration solution according to each candidate configuration solution and the pre-built optimal solution model.

[0112] Among them, the optimal solution model is a pre-built mathematical model based on the characteristics of bioinformatics data processing tasks and the resource characteristics of the cloud computing platform. It establishes the relationship between task requirements and resource supply. It can predict a series of performance indicators corresponding to each plan (such as system throughput, average task processing time, resource utilization, etc.) based on the input candidate configuration plan, thereby providing a scientific basis for screening out the optimal target configuration plan.

[0113] In the embodiment of the present application, first, each candidate configuration scheme is sorted and prepared according to the input format required by the optimal solution model to ensure that the model can accurately identify the key parameters in each candidate scheme, such as the type and number of computing instances, the resource configuration of each instance, and the allocation strategy of tasks on each instance. Then, the sorted candidate configuration schemes are input into the pre-built optimal solution model one by one. The optimal solution model performs performance index prediction processing on each candidate configuration scheme based on the algorithm constructed internally and the relationship model obtained by training based on a large amount of historical data. These performance indicators include but are not limited to system throughput (the number of tasks completed per unit time), average task processing time, resource utilization (computing core utilization, memory utilization, etc.) and cost-effectiveness (such as the task processing volume or data processing results obtained per unit of computing resource input). For example, for a candidate configuration scheme, the model may predict that its system throughput is 50 samples per hour, the average task processing time is 2 hours, the computing core utilization is 75%, the memory utilization is 80%, and the cost-effectiveness is that 1000GB of data can be processed for every 100 yuan spent.

[0114] Next, according to the specific goals and needs of the bioinformatics data processing project, set corresponding weights for each performance indicator. For example, if the main goal of the current project is to quickly complete urgent biological sample analysis tasks (such as viral gene sequencing analysis in response to sudden epidemics), then the weight of the average task processing time may be set higher, such as 0.4; while for long-term large-scale bioinformatics research projects, more emphasis may be placed on the balance between cost-effectiveness and resource utilization, and the cost-effectiveness weight may be set to 0.3, the resource utilization weight to 0.2, the system throughput weight to 0.1, etc. These weights reflect the relative importance of different performance indicators in the context of a specific project.

[0115] Finally, according to the set weights, the comprehensive performance score of each candidate configuration scheme is calculated by weighted summation. For example, the comprehensive performance score of a candidate configuration scheme = system throughput prediction value × throughput weight + task average processing time prediction value × processing time weight + resource utilization prediction value × utilization weight + cost-effectiveness prediction value × cost-effectiveness weight. By comparing the comprehensive performance scores of each candidate configuration scheme, the scheme with the highest score is selected as the target configuration scheme. This target configuration scheme represents the optimal resource configuration choice under the current bioinformatics data processing task requirements, cloud computing platform resource status, and performance and cost considerations, and will be applied in the actual bioinformatics data processing process.

[0116] S504: Perform resource allocation processing on the remaining batches of bioinformatics processing tasks according to the target allocation plan to obtain resource allocation results.

[0117] The embodiment of the present application can adopt a dynamic adjustment strategy when performing resource configuration processing on the remaining batches of bioinformatics processing tasks according to the target configuration scheme. During the execution of the task, continue to use the runtime monitoring module to monitor the execution of the task in real time. If it is found that the current target configuration scheme is no longer the best choice due to changes in data characteristics, fluctuations in the resource status of the cloud computing platform, etc., for example, the processing time of a task begins to increase or the resource utilization rate becomes unreasonable, the target configuration scheme is adjusted in time. The amount of resources required for the task can be recalculated, the thread configuration, computing instance allocation, etc. can be changed to adapt to the new situation, to ensure that the remaining batches of bioinformatics processing tasks are always carried out under the optimal resource configuration, and to obtain resource configuration results that are more in line with the actual situation.

[0118] The embodiment of the present application can realize accurate resource allocation for the remaining batches of bioinformatics processing tasks by accurately calculating the resource requirements of the remaining batches of bioinformatics processing tasks according to the target resource configuration information, and then generating candidate configuration schemes by combining multiple computing instances, and screening out the target configuration scheme using the optimal solution model, and finally performing resource configuration processing according to the target configuration scheme. This accurate configuration avoids over-allocation or under-allocation of resources, ensures that each task can be efficiently executed in a suitable resource environment, and improves resource utilization efficiency and processing efficiency.

[0119] In an exemplary embodiment, based on the above embodiment, see Figure 6 The embodiment of the present application relates to a process of determining a target configuration scheme according to each candidate configuration scheme and a pre-built optimal solution model, including the following steps:

[0120] S601, performing performance indicator prediction processing on each candidate configuration scheme based on the optimal solution model to obtain a performance indicator prediction value corresponding to each candidate configuration scheme.

[0121] Among them, the performance index prediction value is the result obtained after the performance index prediction processing of each candidate configuration scheme through the optimal solution model, including but not limited to the prediction values ​​of system throughput (the number of tasks completed per unit time), average task processing time, resource utilization (computing core utilization, memory utilization, etc.) and cost-effectiveness (such as the task processing volume or data processing results obtained per unit of computing resource input). These prediction values ​​reflect the theoretical performance of each candidate configuration scheme and are an important basis for evaluating the pros and cons of each scheme.

[0122] In the embodiment of the present application, first, it is ensured that the optimal solution model has been trained and is in a usable state. The training process of the model is usually based on a large amount of historical bioinformatics data processing task data and the corresponding resource allocation. By analyzing the relationship between these data, an accurate mathematical relationship between task requirements and resource supply is established.

[0123] Then, each candidate configuration scheme is sorted and prepared according to the input format required by the optimal solution model. This includes clarifying the key parameters in each candidate scheme, such as the type and number of computing instances, the resource configuration of each instance (such as the number of computing cores, memory capacity, etc.), and the task allocation strategy on each instance. For example, a candidate configuration scheme may use 5 computing instances of a certain specification (such as each instance has 8 cores and 16GB of memory), and the remaining batch of bioinformatics processing tasks are allocated to these instances in a certain proportion. Then, when inputting the model, this information needs to be accurately conveyed to the model. Then, the sorted candidate configuration schemes are input one by one into the pre-built optimal solution model. The optimal solution model performs performance indicator prediction processing on each candidate configuration scheme based on its internally constructed algorithm and the relationship model trained based on a large amount of historical data.

[0124] S602: Determine the comprehensive performance score corresponding to each candidate configuration scheme according to the pre-acquired performance indicator weight and the predicted value of each performance indicator.

[0125] Among them, the performance indicator weight is the relative importance coefficient set for each performance indicator according to the specific goals and needs of the bioinformatics data processing project. Different projects may have different focuses on different performance indicators. For example, for urgent clinical sample analysis tasks, more attention may be paid to shortening the average task processing time, and the weight of the average task processing time will be relatively high; while for long-term large-scale bioinformatics research projects, more attention may be paid to the balance between cost-effectiveness and resource utilization, and appropriate weights will be set for these indicators accordingly. The performance indicator weight is used to weight the predicted values ​​of each performance indicator according to its importance when comprehensively evaluating each candidate configuration scheme, so as to obtain a comprehensive performance score that can better reflect the overall advantages and disadvantages of the scheme.

[0126] The comprehensive performance score is a value calculated by weighted summation based on the pre-acquired performance indicator weights and the predicted values ​​of each performance indicator, and is used to comprehensively evaluate the overall performance of each candidate configuration scheme. The higher the comprehensive performance score, the better the candidate configuration scheme performs in meeting project requirements, and it is the key basis for selecting the target configuration scheme.

[0127] In the embodiment of the present application, first, according to the specific goals and needs of the bioinformatics data processing project, corresponding weights are set for various performance indicators. For example, if the main goal of the current project is to quickly complete urgent biological sample analysis tasks (such as viral gene sequencing analysis in response to sudden outbreaks), then the weight of the average task processing time may be set higher, such as 0.4; while for long-term large-scale bioinformatics research projects, more emphasis may be placed on the balance between cost-effectiveness and resource utilization, and the cost-effectiveness weight may be set to 0.3, the resource utilization weight to 0.2, and the system throughput weight to 0.1. These weights reflect the relative importance of different performance indicators in the context of a specific project and are an important basis for comprehensive evaluation.

[0128] Then, the comprehensive performance score of each candidate configuration solution is calculated according to the following weighted sum formula:

[0129] Comprehensive performance score = system throughput prediction value × throughput weight + task average processing time prediction value × processing time weight + resource utilization prediction value × utilization weight + cost-effectiveness prediction value × cost-effectiveness weight.

[0130] For example, the system throughput prediction value of a candidate configuration scheme is 50 (the unit is set according to the actual situation, such as the number of tasks processed per hour), the average task processing time prediction value is 2 (hours), the resource utilization prediction value is 75% (converted to the decimal 0.75), the cost-effectiveness prediction value is 10 (quantitative expression, such as the amount of data that can be processed for every 1 yuan spent), and the throughput weight is 0.2, the processing time weight is 0.3, the utilization weight is 0.2, and the cost-effectiveness weight is 0.3. Then its comprehensive performance score = 50×0.2+2×0.3+0.75×0.2+10×0.3 (the calculation result is obtained based on actual calculation).

[0131] Through this calculation method, the performance of each candidate configuration scheme on different performance indicators is combined into a single score, which facilitates direct comparison of the overall advantages and disadvantages of each scheme.

[0132] S603: Determine the candidate configuration solution with the highest comprehensive performance score as the target configuration solution.

[0133] In the embodiment of the present application, after calculating the comprehensive performance scores of each candidate configuration scheme, these scores are compared. By comparing the comprehensive performance scores of each candidate configuration scheme one by one, the scheme with the highest score is found. For example, after calculation, the comprehensive performance score of candidate configuration scheme A is 80, the comprehensive performance score of candidate configuration scheme B is 75, and the comprehensive performance score of candidate configuration scheme C is 85, then candidate configuration scheme C with the highest comprehensive performance score is determined as the target configuration scheme.

[0134] This target configuration scheme represents the optimal resource configuration choice under the current bioinformatics data processing task requirements, cloud computing platform resource conditions, and performance and cost considerations. It will be applied in the actual bioinformatics data processing process to achieve optimal resource utilization and efficient task execution.

[0135] The embodiment of the present application uses the optimal solution model to perform performance index prediction processing on each candidate configuration scheme, which can fully and accurately understand the performance of each candidate configuration scheme on different performance indicators. Combined with the pre-set performance index weights, the comprehensive performance score is calculated, providing a scientific and reasonable basis for screening the target configuration scheme. This model- and weight-based evaluation method avoids the drawbacks of traditional resource allocation decisions based solely on experience or subjective judgment, and improves the scientificity and accuracy of resource allocation decisions.

[0136] In an exemplary embodiment, based on the above embodiment, the method of the embodiment of the present application includes:

[0137] When the resource demand of the target configuration solution is greater than the available resources of the current platform, the cloud computing platform is expanded. When the resource load value of a single computing instance indicates no load and the bioinformatics processing task corresponding to the single computing instance indicates task completion, the cloud computing platform is reduced.

[0138] Among them, a single computing instance is a basic computing unit with specific computing resource configuration provided by the cloud computing platform, which usually includes a certain number of computing cores, memory capacity, and related network configuration. Each bioinformatics processing task will be assigned to a different single computing instance for execution, and its resource load will change with the execution of the task.

[0139] The current platform available resources refer to the total amount of various resources that are actually available for allocation and use on the cloud computing platform at a certain moment, including the number of idle computing cores, remaining memory capacity, unoccupied storage resources, etc. It is a reflection of the actual ability of the cloud computing platform to provide resource support for tasks.

[0140] The embodiments of the present application may adopt a prediction-based capacity expansion method. By analyzing the historical processing data of the remaining batches of bioinformatics processing tasks and the progress of the current tasks, the changing trend of the resource demand of the tasks in the future (such as the next few hours or a specific stage in the batch processing process) is predicted. For example, based on the resource consumption growth law of similar tasks in the past in a specific link (such as the gene analysis link), combined with the number of samples and processing progress of the current remaining batch tasks in this link, it is predicted that the computing core demand will increase significantly in a certain period of time in the future. Even if the current resource demand has not yet exceeded the current platform available resources, the capacity expansion preparation can be started in advance. In this case, the above-mentioned steps of capacity expansion resource planning and determination and capacity expansion operation execution are also followed, except that the basis for capacity expansion is based on the predicted changes in resource demand, rather than just based on the current resource supply and demand comparison.

[0141] The runtime monitoring module continuously monitors the resource load value of each single computing instance in real time, and monitors the completion status of the bioinformatics processing tasks assigned to each single computing instance. For the monitoring of resource load values, we mainly focus on indicators such as the utilization rate of computing cores and the occupancy rate of memory. When these indicators are displayed at extremely low levels for a continuous period of time (such as 5 minutes) (such as the computing core utilization rate is less than 5%, and the memory occupancy rate is less than 10%), it indicates that the single computing instance is in a no-load state. At the same time, through the task management system or a specific task identification mechanism, it is determined that the bioinformatics processing tasks assigned to the single computing instance have been completed, that is, the conditions for the completion of the bioinformatics processing tasks corresponding to the single computing instance are met. When these two conditions are met at the same time, the scaling-down process is triggered.

[0142] Once the scaling-down condition is triggered, first ensure that all relevant data on the single computing instance has been properly processed to avoid data loss or damage. For example, back up or migrate temporary files and cached data stored on the single computing instance to other storage locations (such as shared storage devices), and then perform the scaling-down operation by calling specific interfaces provided by the cloud computing platform (such as the interface for destroying computing instances). After receiving the scaling-down request, the cloud computing platform will release the hardware resources occupied by the single computing instance (such as recycling the CPU, memory and other resources of the physical server to the resource pool), update the resource status information in the resource management system, mark the single computing instance as destroyed, and remove the relevant information of the single computing instance from the list of available computing instances, thereby realizing the scaling-down of the cloud computing platform.

[0143] The embodiment of the present application can timely replenish the required resources by performing capacity expansion when the resource demand of the target configuration scheme is greater than the available resources of the current platform, ensuring that the bioinformatics data processing task will not stagnate or be inefficient due to insufficient resources, so that resources are fully utilized and the task processing efficiency is improved. At the same time, when a single computing instance is unloaded and the task is completed, capacity reduction is performed to release idle resources and avoid resource waste, further optimizing the resource utilization efficiency of the cloud computing platform, so that resources can be reasonably allocated at different task stages.

[0144] In an exemplary embodiment, based on the above embodiment, see Figure 7 The embodiment of the present application relates to a process of expanding the capacity of a cloud computing platform, including the following steps:

[0145] S701: Determine a computing instance to be added according to resource demand.

[0146] The embodiment of the present application first analyzes in detail the demand for key resource indicators such as the number of computing cores and the total amount of memory in the resource demand. For example, the resource demand determines that 800 computing cores and 3200GB of memory are required to complete the remaining batches of bioinformatics processing tasks. Then, the computing instance information of various specifications provided by the cloud computing platform is obtained. These computing instances have different numbers of cores, amounts of memory, and possible other resource configuration characteristics (such as storage capacity, network bandwidth, etc.). For example, the platform provides the following computing instances: Instance A (40 cores, 160GB memory), Instance B (20 cores, 80GB memory), Instance C (10 cores, 40GB memory).

[0147] Next, based on the resource requirements and the specifications of the computing instances, determine the combination of computing instances to be added through mathematical calculations and resource matching. Taking the requirements for computing cores and memory as an example, the following methods can be used:

[0148] Suppose the number of instances A to be added is x, the number of instances B is y, and the number of instances C is z. Based on the number of computing cores required, the equation can be listed as: 40x+20y+10z=800; based on the total memory requirement, the equation can be listed as: 160x+80y+40z=3200. By solving this set of simultaneous equations (you can use conventional mathematical methods such as substitution and elimination), you can get a set of solutions that meet the conditions, such as x=10, y=20, z=0 (this is just an example, the actual solution may vary depending on the instance specifications and requirements provided by the platform). This means that the computing instances to be added are determined to be 10 instances A and 20 instances B.

[0149] When determining the computing instances to be added, other factors need to be considered, such as cost-effectiveness, resource utilization, and compatibility with existing tasks and resources. For example, although a computing instance can meet the demand in terms of resource matching, if its cost is too high or its resource utilization is low in the past, its usage may be appropriately reduced, and other more cost-effective computing instances may be increased.

[0150] S702: Create a computing instance interface for the cloud computing platform according to the computing instance to be added.

[0151] In the embodiment of the present application, first, for the determined computing instances to be added (taking the 10 instances A and 20 instances B mentioned above as an example), the detailed parameters of the computing instances to be added are sorted out in the format required by the cloud computing platform. These parameters include the number of cores, memory, storage capacity (if applicable), network bandwidth requirements (if applicable), etc. of each instance. For example, for instance A, the detailed information such as the number of cores is 40, the amount of memory is 160GB, the storage capacity is 500GB (assuming), and the network bandwidth requirement is 100Mbps (assuming) is sorted out; for instance B, the corresponding parameter information is sorted out.

[0152] Then, call the specific interface provided by the cloud computing platform (such as the interface for creating computing instances), and pass the sorted detailed parameters of the computing instances to be added as request parameters to the cloud computing platform. After receiving the request, the cloud computing platform allocates corresponding hardware resources (such as the CPU and memory of the physical server) from its resource pool according to the provided parameters. For example, for 10 instances of A, the platform will find enough physical servers and allocate 400 computing cores (10×40) and 1600GB of memory (10×160); for 20 instances of B, 400 computing cores (20×20) and 1600GB of memory (20×80) are allocated.

[0153] Next, the platform performs a series of configuration operations such as virtualization, network configuration, and operating system installation on the allocated resources to convert them into computing instances that can be directly used by bioinformatics processing tasks. In this process, it is ensured that each computing instance has an independent operating environment, including appropriate network settings, operating system versions, etc., so that it can run normally and provide the required computing power and memory space. For example, install a specified version of the operating system for each instance A, configure the network so that it can communicate with other computing instances and external networks, set memory management policies, etc.; similarly, perform corresponding configuration operations on instance B.

[0154] Finally, after completing the creation of the computing instance interface, the newly created computing instance is integrated into the existing cloud computing platform environment so that it can work with the original computing resources to provide additional computing power and resource support for the remaining batches of bioinformatics processing tasks. For example, the resource management system of the cloud computing platform is updated, the newly created computing instance is added to the list of available computing instances, and the task scheduling strategy is adjusted so that the remaining batches of bioinformatics processing tasks can be reasonably allocated to the new and old computing instances to ensure that the tasks can be efficiently executed in the expanded resource environment.

[0155] The embodiment of the present application can accurately expand the resources of the cloud computing platform by accurately determining the computing instances to be added according to the resource demand, avoiding the waste of resources or insufficient resources caused by blindly adding computing instances, ensuring that the newly added computing instances can just meet the needs of bioinformatics data processing tasks, and improving resource utilization efficiency.

[0156] In an exemplary embodiment, based on the above embodiment, the method of the embodiment of the present application further includes the following steps:

[0157] Step 1: Generate multiple batches of bioinformatics processing tasks based on the pre-acquired bioinformatics raw data; determine the job distribution resources corresponding to the first batch of bioinformatics processing tasks based on the pre-established basic experience strategy set and the pre-acquired bioinformatics operation process;

[0158] Step 2: According to the task processing data of each sample in the first batch of bioinformatics processing tasks, multiple multi-process single-Mega-data processing times are determined; the number of processes corresponding to each multi-process single-Mega-data processing time is different; each multi-process single-Mega-data processing time is normalized to obtain multiple single-process single-Mega-data processing times; the resource configuration information with the smallest single-process single-Mega-data processing time is determined as the target resource configuration information;

[0159] Step 3: Determine the resource requirements of the remaining batches of bioinformatics processing tasks according to the target resource configuration information; determine multiple candidate configuration schemes according to the resource requirements and multiple pre-acquired computing instances;

[0160] Step 4: Based on the optimal solution model, the performance indicator prediction processing is performed on each candidate configuration scheme to obtain the performance indicator prediction value corresponding to each candidate configuration scheme; according to the pre-acquired performance indicator weight and each performance indicator prediction value, the comprehensive performance score corresponding to each candidate configuration scheme is determined; the candidate configuration scheme with the highest comprehensive performance score is determined as the target configuration scheme;

[0161] Step 5: Perform resource allocation processing on the remaining batches of bioinformatics processing tasks according to the target allocation plan to obtain resource allocation results;

[0162] Step 6: When the resource demand of the target configuration solution is greater than the available resources of the current platform, determine the computing instances to be added based on the resource demand; create computing instance interfaces for the cloud computing platform based on the computing instances to be added; and scale down the cloud computing platform when the resource load value of a single computing instance indicates no load and the bioinformatics processing task corresponding to the single computing instance indicates task completion.

[0163] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0164] Based on the same inventive concept, the embodiment of the present application also provides a resource configuration device for implementing the resource configuration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more resource configuration device embodiments provided below can refer to the limitations on the resource configuration method above, and will not be repeated here.

[0165] In one embodiment, Figure 8 As shown, a resource configuration device 800 is provided, comprising:

[0166] The task generation module 801 is used to generate multiple batches of bioinformatics processing tasks based on the pre-acquired bioinformatics raw data;

[0167] The analysis module 802 is used to perform configuration analysis on the first batch of bioinformatics processing tasks to obtain target resource configuration information of the bioinformatics processing tasks;

[0168] The configuration module 803 is used to perform resource configuration processing on the remaining batches of bioinformatics processing tasks according to the target resource configuration information to obtain resource configuration results corresponding to each bioinformatics processing task.

[0169] In one embodiment, the analysis module includes:

[0170] A resource determination unit, used to determine the job distribution resources corresponding to the first batch of bioinformatics processing tasks based on a pre-established basic experience strategy set and a pre-acquired bioinformatics job flow;

[0171] A task execution unit is used to execute the first batch of bioinformatics processing tasks according to the job distribution resources, and obtain the task processing data of each sample in the first batch of bioinformatics processing tasks;

[0172] The information determination unit is used to determine the target resource configuration information according to the task processing data of each sample in the first batch of bioinformatics processing tasks.

[0173] In one embodiment, the information determining unit includes:

[0174] The time determination subunit is used to determine multiple multi-process single-Megabyte data processing times according to the task processing data of each sample in the first batch of bioinformatics processing tasks; the number of processes corresponding to each multi-process single-Megabyte data processing time is different;

[0175] A normalization subunit is used to normalize the single-Mega-data processing time of each multi-process to obtain multiple single-process single-Mega-data processing times;

[0176] The information determination subunit is used to determine the resource configuration information with the minimum processing time for single-process single-M data as the target resource configuration information.

[0177] In one embodiment, the configuration module includes:

[0178] A demand determination unit, used to determine the resource demand of the remaining batches of bioinformatics processing tasks according to the target resource configuration information;

[0179] A candidate solution determination unit, used to determine multiple candidate configuration solutions according to resource requirements and multiple pre-acquired computing instances;

[0180] A target solution determination unit, used to determine a target configuration solution according to each candidate configuration solution and a pre-built optimal solution model;

[0181] The resource configuration unit is used to perform resource configuration processing on the remaining batches of bioinformatics processing tasks according to the target configuration plan to obtain resource configuration results.

[0182] In one embodiment, the target solution determination unit includes:

[0183] A prediction subunit is used to perform performance indicator prediction processing on each candidate configuration scheme based on the optimal solution model to obtain a performance indicator prediction value corresponding to each candidate configuration scheme;

[0184] The score determination subunit is used to determine the comprehensive performance score corresponding to each candidate configuration scheme according to the pre-acquired performance indicator weight and the predicted value of each performance indicator;

[0185] The target solution determination subunit is used to determine the candidate configuration solution with the highest comprehensive performance score as the target configuration solution.

[0186] In one embodiment, the above device comprises:

[0187] The capacity expansion module is used to expand the capacity of the cloud computing platform when the resource demand of the target configuration solution is greater than the available resources of the current platform;

[0188] The capacity reduction module is used to reduce the capacity of the cloud computing platform when the resource load value of a single computing instance indicates no load and the bioinformatics processing task corresponding to the single computing instance indicates task completion.

[0189] In one embodiment, the expansion module includes:

[0190] An instance determination unit, used to determine a computing instance to be added according to resource demand;

[0191] The interface creation unit is used to perform computing instance interface creation processing on the cloud computing platform according to the computing instance to be added.

[0192] Each module in the resource configuration device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0193] In one embodiment, a computer device is provided, which may be a public cloud, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a resource configuration method is implemented.

[0194] Those skilled in the art will understand that Fig. 9The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0195] According to some embodiments of the present application, a computer device including instructions is also provided, such as a memory and a processor including instructions. The instructions can be executed by the processor of the computer device to complete the above method.

[0196] According to some embodiments of the present application, a computer program product is also provided, and when the computer program is executed by a processor, the above method can be implemented. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, part or all of the above method can be implemented in whole or in part according to the process or function described in the embodiment of the present application.

[0197] According to some embodiments of the present application, there is also provided a non-transitory computer-readable storage medium including instructions, such as a memory including instructions, and the above instructions can be executed by a processor of an electronic device to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0198] According to some embodiments of the present application, a computer program product is also provided, and when the computer program is executed by a processor, the above method can be implemented. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, part or all of the above method can be implemented in whole or in part according to the process or function described in the embodiment of the present application.

[0199] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0200] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0201] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0202] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A resource configuration method, characterized in that: The method comprises: Generate multiple batches of bioinformatics processing tasks based on pre-acquired bioinformatics raw data; Performing configuration analysis on the first batch of bioinformatics processing tasks to obtain target resource configuration information of the bioinformatics processing tasks; Resource configuration processing is performed on the remaining batches of bioinformatics processing tasks according to the target resource configuration information to obtain resource configuration results corresponding to each of the bioinformatics processing tasks.

2. The method according to claim 1, characterized in that The configuration analysis and processing of the first batch of bioinformatics processing tasks to obtain target resource configuration information of the bioinformatics processing tasks includes: Determine the job distribution resources corresponding to the first batch of bioinformatics processing tasks based on a pre-established basic experience strategy set and a pre-acquired bioinformatics job process; Execute the first batch of bioinformatics processing tasks according to the job distribution resources, and obtain task processing data of each sample in the first batch of bioinformatics processing tasks; The target resource configuration information is determined according to the task processing data of each sample in the first batch of bioinformatics processing tasks.

3. The method according to claim 2, characterized in that The determining target resource configuration information according to the task processing data of each sample in the first batch of bioinformatics processing tasks includes: Determine a plurality of multi-process single-Megabyte data processing times according to the task processing data of each sample in the first batch of bioinformatics processing tasks; each of the multi-process single-Megabyte data processing times corresponds to a different number of processes; Normalizing the multi-process single-Megabyte data processing time to obtain multiple single-process single-Megabyte data processing times; The resource configuration information with the minimum single-process single-M data processing time is determined as the target resource configuration information.

4. The method according to claim 1, characterized in that: The performing resource configuration processing on the remaining batches of bioinformatics processing tasks according to the target resource configuration information to obtain resource configuration results corresponding to each of the bioinformatics processing tasks includes: Determine the resource requirements of the remaining batches of bioinformatics processing tasks according to the target resource configuration information; Determine multiple candidate configuration solutions according to the resource demand and multiple pre-acquired computing instances; Determine a target configuration scheme according to each of the candidate configuration schemes and the pre-built optimal solution model; Resource allocation processing is performed on the remaining batches of bioinformatics processing tasks according to the target allocation scheme to obtain the resource allocation result.

5. The method according to claim 4, characterized in that Determining a target configuration scheme according to each of the candidate configuration schemes and the pre-built optimal solution model includes: Performing performance indicator prediction processing on each of the candidate configuration schemes based on the optimal solution model to obtain a performance indicator prediction value corresponding to each of the candidate configuration schemes; Determine the comprehensive performance score corresponding to each of the candidate configuration schemes according to the pre-acquired performance indicator weights and the predicted values ​​of each of the performance indicators; The candidate configuration scheme with the highest comprehensive performance score is determined as the target configuration scheme.

6. The method according to claim 4, characterized in that The method further comprises: When the resource demand of the target configuration solution is greater than the available resources of the current platform, the cloud computing platform is expanded; When the resource load value of a single computing instance indicates no load and the bioinformatics processing task corresponding to the single computing instance indicates task completion, the cloud computing platform is scaled down.

7. The method according to claim 6, characterized in that The processing of expanding the capacity of the cloud computing platform includes: Determine the computing instance to be added according to the resource demand; The computing instance interface creation process is performed on the cloud computing platform according to the computing instance to be added.

8. A resource allocation device, characterized in that: The device comprises: A task generation module is used to generate multiple batches of bioinformatics processing tasks based on pre-acquired bioinformatics raw data; An analysis module is used to perform configuration analysis on the first batch of bioinformatics processing tasks to obtain target resource configuration information of the bioinformatics processing tasks; A configuration module is used to perform resource configuration processing on the remaining batches of bioinformatics processing tasks according to the target resource configuration information to obtain resource configuration results corresponding to each of the bioinformatics processing tasks.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images