Method and system for optimizing read-write performance of storage unit

Abstract

The invention relates to the technical field of read-write performance optimization. The invention relates to a method and a system for optimizing read-write performance of a storage unit. The method comprises the following steps: S1, obtaining performance parameters of a storage unit, carrying out modular partitioning on the storage unit, obtaining a task issuing area, a data storage area and a temporary management area, carrying out performance resource allocation on each area according to the performance parameters, and meanwhile, reserving idle performance resources; according to the method, the matching degree of the storage area and the data is greatly improved through a modularized intelligent partition and data accurate adaptation mechanism, the performance bottleneck caused by traditional partition stiffness is solved from the source, and based on core performance parameters such as the read-write speed, the bandwidth and the capacity of the storage unit, the storage capacity of the storage unit is improved. Functional modules such as a task issuing area, a data storage area and a temporary management area are divided, exclusive area portraits are constructed for all sub-storage areas, data labels are extracted through stage sampling, and accurate association of data and the storage areas is achieved through weighted adaptation analysis.

Description

Technical Field

[0001] This invention relates to the field of read / write performance optimization technology, and more specifically, to a method and system for optimizing the read / write performance of a storage unit. Background Technology

[0002] As the core hardware carrier for data storage and processing, the read and write performance of storage units directly determines the operating efficiency of the entire data system. Optimizing the read and write performance of storage units is a key technological support in the field of data storage.

[0003] Currently, storage unit read / write performance optimization technologies are mainly applied in large-scale data storage scenarios in data centers. For massive amounts of structured and unstructured data, technologies such as array storage and distributed caching are used to improve concurrent read / write capabilities. However, rigid partition management and the use of fixed physical or logical partitioning models in existing technologies fail to dynamically adapt to data characteristics and storage unit performance parameters. This results in high-frequency access data being mixed with low-frequency archived data, leading to low matching between storage areas and data. When read / write requests are concentrated, performance bottlenecks easily form. Furthermore, data transmission and adaptation lack closed-loop optimization. Existing technologies often perform batch processing after data is fully received, failing to anticipate data needs through staged sampling and real-time adaptation analysis. They also lack secondary verification and adjustment mechanisms for transmitted data, easily leading to data mismatch and decreased read / write efficiency. Therefore, this paper proposes a storage unit read / write performance optimization method. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for optimizing the read and write performance of storage units, so as to solve the problems mentioned in the background art.

[0005] To address the aforementioned technical problems, one objective of this invention is to provide a method for optimizing the read / write performance of a storage unit, comprising the following steps: S1. Obtain the performance parameters of the storage unit, perform modular partitioning of the storage unit to obtain the task release area, data storage area and temporary management area, and allocate performance resources to each area according to the performance parameters, while reserving idle performance resources. S2, Task Release Area: Monitors the receiving module of the storage unit, sets the stage length, samples the transmission data obtained by the receiving module in stages, and releases the transmission task in stages according to the data tags corresponding to the stage data obtained by stage sampling. S3, Data storage area: Based on the initial data in the data storage area, obtain the corresponding regional profile, perform regional adaptation analysis based on the regional profile and data tags, and generate sub-regions and transfer initial data according to the analysis results until the matching degree meets the transmission task, then accept the corresponding stage of the transmission task. S4, Temporary Management Area: Monitors the transmitted data corresponding to the transmission task in the data storage area, performs individual adaptation analysis on the transmitted data and the regional profile, stops and transfers the untransmitted data of the transmission task based on the analysis results, and updates the transmission task in the task release area based on the bytes of the transmitted data. S5. Dynamically allocate idle performance resources to the task release area, data storage area, and temporary management area.

[0006] Preferably, in S1, the performance parameters include the read / write speed of the storage unit, the storage capacity, and the data transmission bandwidth; Modular partitioning of storage units based on performance parameters; The task publishing area is configured to prioritize the use of the fastest read / write portion of the storage unit; The data storage area is configured to occupy the main storage capacity of the storage unit; The temporary management area is configured to have low latency and high throughput for caching and temporary data processing.

[0007] Preferably, in step S1, the storage unit is used as a resource pool based on performance data. At the same time, when initializing and setting the task release area, data storage area and temporary management area, performance resources are fixedly allocated to each area, and idle performance resources are reserved in the resource pool for dynamic resource allocation. Each area can work independently. The data transmission order is from the task release area to the data storage area, then from the data storage area to the temporary management area, and finally from the temporary management area back to the task release area.

[0008] Preferably, in step S2, a monitoring linkage is established with the receiving module of the storage unit to obtain the transmission data received by the receiving module; secondly, the stage length is set according to the historical data transmission volume and the performance resources allocated in the task release area, and then sampling is performed according to the stage length.

[0009] Preferably, the sampling step based on the stage length is as follows: S2.1 When the receiving module detects that a transmission data input has been generated, the transmission data is sampled in stages according to the set stage length, so as to obtain the stage data corresponding to the transmission data sampling. Each stage of data corresponds to the transmission data of its respective stage length; After the transmitted data was detected, phased sampling was performed in a timely manner. However, the transmitted data was not fully received during this process. S2.2 Perform data label analysis on the data of each stage to obtain the data labels corresponding to the data of each stage; S2.3. Based on the data tags of the data at each stage, release transmission tasks to form a transmission task set; The transmission task includes stage data and the transmission data within the corresponding stage. The number of transmission tasks is rounded up to the nearest integer.

[0010] Preferably, in step S3, the data storage area includes multiple sub-storage areas, and the specific steps are as follows: S3.1. Generate a region profile based on the initial data of the sub-storage area and obtain the region profile corresponding to the sub-storage area; S3.2 Perform region adaptation analysis on the region profile of each sub-storage area and the data label corresponding to the transmission task to obtain the adaptation degree between each sub-storage area and the transmission task. Then select the sub-storage area with the highest adaptation degree to accept the transmission task and form the transmission association between the sub-storage area and the transmission task. S3.2.1 Set the adaptation threshold based on performance parameters; S3.2.2 Compare the fit between the transmission task and the sub-storage area with the fit threshold. If the fit is less than the fit threshold, cancel the transmission association between the sub-storage area and the transmission task, and create a new sub-storage area with the data tag of the transmission task. Then, associate the newly created sub-storage area with the transmission task. Conversely, if the fit is greater than the fit threshold, monitoring will continue.

[0011] Preferably, in step S4, the temporary management area temporarily stores the untransmitted task data, and the specific steps are as follows: S4.1 The temporary management area extracts the transmitted data of the corresponding transmission task from the sub-storage area, performs individual adaptation analysis on the transmitted data and the regional profile of the corresponding sub-storage area, obtains the adaptation degree between the transmitted data and the sub-storage area, and compares it with the adaptation threshold. S4.2 is only performed when the adaptation degree of the comparison result is less than the adaptation threshold. S4.2. Pause the transmission tasks corresponding to the transmitted data whose adaptability is less than the adaptability threshold, and gradually transfer the transmitted data until the transmitted data in the sub-storage area meets the adaptability threshold. When the transmitted data is transferred, it is restored to the untransmitted data of the transmission task, and the untransmitted data of the transmission task is transferred to the temporary management area; S4.3 Calculate the bytes of data already transmitted in the sub-storage area, and perform a second stage division on the untransmitted data of the same transmission task based on the bytes of the transmitted data to obtain the untransmitted data of the second stage division. Then, the task release area performs data tag analysis and task release on the untransmitted data of the second stage division.

[0012] Preferably, in step S5, corresponding idle performance resources are prepared according to the amount of stage data, and the task release area, data storage area and temporary management area are dynamically scheduled. Set a scheduling resource threshold. When the remaining available performance resources in any of the task release area, data storage area, and temporary management area are lower than the scheduling resource threshold, the corresponding idle performance resources will be used to supplement the performance resources.

[0013] The second objective of this invention is to provide a storage unit read / write performance optimization system, including any one of the storage unit read / write performance optimization methods described above, comprising a performance management module, a task publishing module, a data storage module, and a temporary management module; The performance management module is used to obtain the performance parameters of the storage unit, perform modular partitioning of the storage unit to obtain the task release area, data storage area and temporary management area, and allocate performance resources to each area according to the performance parameters, while reserving idle performance resources and dynamically allocating idle performance resources to the task release area, data storage area and temporary management area. The task publishing module includes a task publishing area, a receiving module for monitoring and storing the storage unit, a phase length for sampling the transmission data obtained by the receiving module in stages, and a phase publishing of transmission tasks based on the data tags corresponding to the phase data obtained by the phase sampling. The data storage module includes a data storage area. Based on the initial data in the data storage area, a corresponding regional profile is obtained. Based on the regional profile and data tags, a regional adaptation analysis is performed. Based on the analysis results, sub-regions and initial transfer data are generated until the matching degree meets the transmission task. Then, the transmission task at the corresponding stage is accepted. The temporary management module includes a temporary management area. It monitors the transmitted data corresponding to the transmission task in the data storage area, performs individual adaptation analysis on the transmitted data and the regional profile, stops and transfers the untransmitted data of the transmission task based on the analysis results, and updates the transmission task in the task publishing area based on the bytes of the transmitted data.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This storage unit read / write performance optimization method and system significantly improves the matching degree between storage areas and data through modular intelligent partitioning and precise data adaptation mechanism, fundamentally solving the performance bottleneck caused by the rigidity of traditional partitioning. Based on the core performance parameters of storage units such as read / write speed, bandwidth, and capacity, it divides functional modules such as task release area, data storage area, and temporary management area, and builds a dedicated area profile for each sub-storage area. Through stage sampling to extract data tags, and through weighted adaptation analysis, it achieves precise association between data and storage areas. At the same time, it supports the creation and dynamic transfer of sub-areas with mismatched data, avoids read / write conflicts caused by mixed data storage, and significantly improves the read / write response speed and concurrent processing capability of storage units.

[0015] 2. This storage unit read / write performance optimization method and system achieves optimal allocation of performance resources, completely changing the resource waste problem of the traditional static allocation mode. It integrates storage unit performance resources into a resource pool, allocates basic fixed resources to each module during initialization and reserves idle resources. By setting differentiated scheduling thresholds, it monitors the remaining available resources of each module in real time. When the resources of any module are lower than the threshold, idle resources are dynamically scheduled to supplement them, ensuring the resource supply for critical task release, data transmission and other links. At the same time, the resource recycling mechanism realizes the recycling of idle resources, which greatly improves the utilization rate of core performance resources such as bandwidth and read / write speed. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a method for optimizing the read / write performance of a storage unit according to the present invention. Figure 2 This is a flowchart of the present invention when the receiving module detects that a transmission data input has been generated; Figure 3 This is a flowchart illustrating the process of generating a region profile based on the initial data of the sub-storage area according to the present invention. Figure 4 This is a flowchart illustrating the process of retrieving transmitted data from the sub-storage area of ​​the temporary management area in this invention. Figure 5 This is a schematic diagram of the overall unit of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 - Figure 5 As shown, the purpose of this embodiment is to provide a method for optimizing the read and write performance of a storage unit, including the following steps: S1. Obtain the performance parameters of the storage unit, perform modular partitioning of the storage unit to obtain the task release area, data storage area and temporary management area, and allocate performance resources to each area according to the performance parameters, while reserving idle performance resources. In S1, performance parameters include the read / write speed of the storage unit, storage capacity, and data transmission bandwidth; Modular partitioning of storage units based on performance parameters (to avoid performance loss caused by functional overlap); The task distribution area is configured to prioritize the use of the fastest read / write speeds in the storage units to ensure low latency in task distribution. The data storage area is configured to occupy the main storage capacity of the storage unit and is used for persistent storage of various types of data; The temporary management area is configured to have low latency and high throughput for caching and temporary data processing.

[0019] In S1, the storage unit is used as a resource pool based on performance data. At the same time, when the task release area, data storage area and temporary management area are initialized, performance resources are fixedly allocated to each area, and idle performance resources are reserved in the resource pool for dynamic resource allocation. Each area can work independently. Reserve 10% to 15% of the total resources as idle resources for subsequent dynamic scheduling (to cope with sudden loads). The data transmission order is from the task release area to the data storage area, then from the data storage area to the temporary management area, and finally from the temporary management area back to the task release area.

[0020] S2, Task Release Area: Monitors the receiving module of the storage unit, sets the stage length, samples the transmission data obtained by the receiving module in stages, and releases the transmission task in stages according to the data tags corresponding to the stage data obtained by stage sampling. In S2, a monitoring linkage is established with the receiving module of the storage unit to obtain the transmitted data received by the receiving module; a two-way communication linkage is established with the receiving module of the storage unit to enable the data input monitoring mode, and the transmitted data being received by the receiving module is captured in real time without waiting for the data to be fully received, thus reducing latency; Secondly, the stage length is set based on historical data transmission volume and the performance resources allocated in the task release area, and then sampling is performed based on the stage length. Combining historical data transmission volume and the performance resources allocated in the task release area, the stage length is calculated and set (balancing sampling representativeness and processing efficiency, avoiding sample bias due to being too short and increased latency due to being too long), as shown in the following formula: ; in, The length of the stage. This represents the historical average data volume transmitted per batch. For adjustment coefficients, Available bandwidth allocated to the task publishing area.

[0021] The steps for sampling based on the stage length are as follows: S2.1 When the receiving module detects that a transmission data input has been generated, the transmission data is sampled in stages according to the set stage length, so as to obtain the stage data corresponding to the transmission data sampling. Each stage of data corresponds to the transmission data of its respective stage length; After the transmitted data is detected, phased sampling is performed immediately. This process does not involve receiving all the transmitted data, and there is no need to wait for the entire batch of data to be received. The coverage of data for each stage: A single piece of stage data precisely corresponds to the transmitted data within a stage length, ensuring that the stage mapping relationship between the sampled data and the original transmitted data is unique.

[0022] S2.2 Perform data label analysis on the data of each stage to obtain the data labels corresponding to the data of each stage; Data tags include data type tags (such as text / image / video), data priority tags (high / medium / low), and data volume tags (single-stage data size). S2.3. Based on the data tags of the data at each stage, release transmission tasks to form a transmission task set; Based on the tag combination, a targeted transmission task is generated. Each task contains two core components: the stage data (sampled sample) of the corresponding stage and the complete transmission data that was not fully received within the length of that stage (ensuring that the task covers the full amount of data). The transmission task includes stage data and the transmission data within the corresponding stage. The number of transmission tasks is rounded up to the nearest integer to avoid data loss.

[0023] S3, Data storage area: Based on the initial data in the data storage area, obtain the corresponding regional profile, perform regional adaptation analysis based on the regional profile and data tags, and generate sub-regions and transfer initial data according to the analysis results until the matching degree meets the transmission task, then accept the corresponding stage of the transmission task. In S3, the data storage area contains multiple sub-storage areas. The initial sub-storage areas can be created based on the stage data of the initial transmission task. A separate sub-storage area is created for each stage of data for data transmission. The specific steps are as follows: S3.1. Generate a region profile based on the initial data of the sub-storage area and obtain the region profile corresponding to the sub-storage area; For each sub-storage area, four core features are extracted through initial data parsing to construct a unique area profile. The area profile includes data type features, data volume features, access characteristics features, and storage performance features.

[0024] S3.2 Perform region adaptation analysis on the region profiles of each sub-storage area and the data tags corresponding to the transmission tasks to obtain the adaptation degree between each sub-storage area and the transmission tasks. Then, select the sub-storage area with the highest adaptation degree to accept the transmission tasks, forming a transmission association between the sub-storage areas and the transmission tasks. The steps for calculating the adaptation degree are as follows: The weights of the four feature classes are determined to highlight the core adaptation dimension. Then, for each data label of the transmission task (corresponding one-to-one with the profile features), the single feature matching degree is calculated (for positive features: the closer the values, the higher the matching degree; for negative features: such as latency, the closer the values, the higher the matching degree). The single feature matching degrees are weighted and summed to obtain the total adaptation degree between the sub-storage area and the transmission task, as shown in the following formula: ; in, Let Z represent the fit between the sub-storage area and the transmission task, and T represent the transmission task. For data type matching degree, For storage performance matching, For data volume matching degree, For access feature matching degree, , , , For feature weights; S3.2.1. Set adaptation thresholds based on performance parameters; Calculate dynamic adaptation thresholds based on the performance parameters of sub-storage areas (such as read / write speed and bandwidth) and business requirements (such as real-time requirements of transmission tasks) to avoid adaptation deviations caused by fixed thresholds, as shown in the following formula: ; in, To dynamically adapt the threshold, Based on the threshold, This represents the maximum read speed of the storage unit. The maximum bandwidth of the storage unit. This represents the priority of the transmission task, with a value range of [0,1]. and For adjustment coefficients, The maximum read speed of the z-th sub-memory area. Allocate bandwidth for the z-th sub-memory area; By extracting three-dimensional features of the transmission task (business importance, real-time requirements, and resource consumption intensity), such as quantification based on business attributes in data tags (core business assigned 1.0, important business assigned 0.8, routine business assigned 0.5, and archiving business assigned 0.2); quantification based on timeliness requirements in data tags (latency tolerance ≤10ms assigned 1.0, ≤100ms assigned 0.8, ≤1s assigned 0.5, and >1s assigned 0.2); and quantification based on data volume and transmission rate requirements in data tags (data volume >1...). Data volumes of 0GB and speed requirements > 1GB / s are assigned a value of 1.0; data volumes of 5-10GB and speed requirements of 0.5-1GB / s are assigned a value of 0.7; data volumes of 1-5GB and speed requirements of 0.1-0.5GB / s are assigned a value of 0.4; and data volumes < 1GB and speed requirements < 0.1GB / s are assigned a value of 0.1. Then, all three dimensions are linearly normalized, and weights are assigned based on requirements, such as a business importance weight of 0.4, a real-time requirement weight of 0.4, and a resource consumption intensity weight of 0.2. Finally, a weighted sum is obtained. .

[0025] S3.2.2 Compare the fit between the transmission task and the sub-storage area with the fit threshold. If the fit is less than the fit threshold, cancel the transmission association between the sub-storage area and the transmission task, and create a new sub-storage area with the data tag of the transmission task. Then, associate the newly created sub-storage area with the transmission task. Conversely, if the fit is greater than the fit threshold, monitoring will continue.

[0026] During the transmission task execution, the regional profile of the sub-storage area is updated in real time (such as the proportion of newly added data types and changes in access frequency) to ensure the accuracy of subsequent task adaptation.

[0027] S4, Temporary Management Area: Monitors the transmitted data corresponding to the transmission task in the data storage area, performs individual adaptation analysis on the transmitted data and the regional profile, stops and transfers the untransmitted data of the transmission task based on the analysis results, and updates the transmission task in the task release area based on the bytes of the transmitted data. In S4, the temporary management area temporarily stores untransmitted task data. The specific steps are as follows: S4.1 The temporary management area extracts the transmitted data of the corresponding transmission task from the sub-storage area, performs individual adaptation analysis on the transmitted data and the regional profile of the corresponding sub-storage area, obtains the adaptation degree between the transmitted data and the sub-storage area, and compares it with the adaptation threshold. S4.2 is only performed when the adaptation degree of the comparison result is less than the adaptation threshold. When performing individual fitting analysis, the fitting analysis is still based on 4 types of features, the same as the S3.1 method; S4.2 Suspend the transmission tasks corresponding to the transmitted data whose fit is less than the fit threshold (to avoid more mismatched data occupying the sub-storage area resources), and gradually transfer the transmitted data until the transmitted data in the sub-storage area meets the fit threshold. Gradual transfer mechanism: Mismatched data that has been transferred in the sub-storage area is gradually transferred to the temporary management area in batches, with the batch size equal to the optimal value of a single read / write operation in the sub-storage area (to avoid performance fluctuations in the sub-storage area caused by a one-time transfer). When the transmitted data is transferred, it is restored to the untransmitted data of the transmission task, and the untransmitted data of the transmission task is transferred to the temporary management area; S4.3 Calculate the bytes of data already transmitted in the sub-storage area, and perform a second stage division on the untransmitted data of the same transmission task based on the bytes of the transmitted data to obtain the untransmitted data of the second stage division. Then, the task release area performs data tag analysis and task release on the untransmitted data of the second stage division.

[0028] The stage data after the secondary partitioning is fed back to the task release area, where data tag analysis (based on the actual characteristics of the latest stage data) is re-executed and transmission task release is carried out. More suitable sub-storage areas are matched for data that has not been transmitted (forming a closed loop of adaptation error → error correction → re-adaptation). The formula for the secondary stage length is as follows: ; in, The single-stage length of the quadratic partition. The length of the original stage of the transmission task. This represents the total number of bytes of data transmitted. This represents the original total number of bytes for the transmission task. The maximum bandwidth of the storage unit. This represents the currently available bandwidth in the temporary management area.

[0029] S5. Dynamically allocate idle performance resources to the task release area, data storage area, and temporary management area.

[0030] In S5, corresponding idle performance resources are prepared based on the amount of data in each stage, and the task release area, data storage area and temporary management area are dynamically scheduled. Set a scheduling resource threshold. When the remaining available performance resources in any of the task release area, data storage area, and temporary management area are lower than the scheduling resource threshold, the corresponding idle performance resources will be supplemented. The specific steps are as follows: The system collects the amount of stage data and the average resource requirement of a single stage data in real time. Then, based on the amount of stage data and the average resource requirement, combined with the upper limit of the total idle resources of the storage unit, it determines the actual amount of idle performance resources to be prepared (ensuring that the preparation amount can cover the peak demand without occupying too much total resources). The reserved idle resources are stored in the resource pool according to three categories: "bandwidth, read / write speed, and capacity", and marked as "resources to be scheduled". They are only used to supplement the three regions and do not participate in other static allocations. Next, the fixed performance resources allocated in S1 for the task release area, data storage area, and temporary management area are obtained as the basis for threshold setting. Different threshold coefficients are allocated according to the differences in the core functions of each area. The more core the function and the higher the sensitivity to performance, the smaller the coefficient (the lower the threshold, the easier it is to trigger replenishment). By multiplying the fixed resource base by the threshold coefficient, the remaining available resource threshold value of each area is obtained (replenishment is triggered when the value is lower than this value). Resources to be scheduled are allocated according to the priority of temporary management area > task release area > data storage area (the temporary management area is responsible for data caching and feedback, and is the key to the closed loop, so it is given priority). When the area that is supplementing resources completes its current transmission task and the remaining available resources rise back to the scheduling threshold plus safety redundancy, the excess resources are reclaimed to the idle resource pool and remarked as resources to be scheduled, for subsequent supplementation by other areas, forming a closed loop of preparation-supplementation-reclaiming-reuse.

[0031] The second objective of this invention is to provide a storage unit read / write performance optimization system, including a storage unit read / write performance optimization method as described above, comprising a performance management module, a task publishing module, a data storage module, and a temporary management module; The performance management module is used to obtain the performance parameters of the storage unit, perform modular partitioning of the storage unit to obtain the task deployment area, data storage area and temporary management area, and allocate performance resources to each area according to the performance parameters, while reserving idle performance resources and dynamically allocating idle performance resources to the task deployment area, data storage area and temporary management area. The task publishing module includes a task publishing area, a receiving module for monitoring storage units, a phase length for sampling the transmission data obtained by the receiving module in stages, and a phase publishing of transmission tasks based on the data tags corresponding to the phase data obtained by the phase sampling. The data storage module includes a data storage area. Based on the initial data in the data storage area, it obtains the corresponding regional profile, performs regional adaptation analysis based on the regional profile and data tags, and generates sub-regions and initial transfer data based on the analysis results until the matching degree meets the transmission task, at which point it accepts the corresponding stage of the transmission task. The temporary management module includes a temporary management area. It monitors the transmitted data corresponding to the transmission task in the data storage area, performs individual adaptation analysis on the transmitted data and the regional profile, stops and transfers the untransmitted data of the transmission task based on the analysis results, and updates the transmission task in the task publishing area based on the bytes of the transmitted data.

[0032] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for optimizing the read / write performance of a storage unit, characterized in that: Includes the following steps: S1. Obtain the performance parameters of the storage unit, perform modular partitioning of the storage unit to obtain the task release area, data storage area and temporary management area, and allocate performance resources to each area according to the performance parameters, while reserving idle performance resources. S2, Task Release Area: Monitors the receiving module of the storage unit, sets the stage length, samples the transmission data obtained by the receiving module in stages, and releases the transmission task in stages according to the data tags corresponding to the stage data obtained by stage sampling. S3, Data storage area: Obtain the corresponding regional profile based on the initial data in the data storage area, perform regional adaptation analysis based on the regional profile and data tags, and generate sub-regions and transfer initial data based on the analysis results until the matching degree meets the transmission task, then accept the transmission task of the corresponding stage. S4, Temporary Management Area: Monitors the transmitted data corresponding to the transmission task in the data storage area, performs individual adaptation analysis on the transmitted data and the regional profile, stops and transfers the untransmitted data of the transmission task based on the analysis results, and updates the transmission task in the task release area based on the bytes of the transmitted data. S5. Dynamically allocate idle performance resources to the task release area, data storage area, and temporary management area.

2. The method for optimizing the read / write performance of a storage unit according to claim 1, characterized in that: In S1, the performance parameters include the read / write speed of the storage unit, the storage capacity, and the data transmission bandwidth. Modular partitioning of storage units based on performance parameters; The task publishing area is configured to prioritize the use of the fastest read / write portion of the storage unit; The data storage area is configured to occupy the main storage capacity of the storage unit; The temporary management area is configured to have low latency and high throughput for caching and temporary data processing.

3. The method for optimizing the read / write performance of a storage unit according to claim 2, characterized in that: In S1, the storage unit is used as a resource pool based on performance data. At the same time, when initializing the task release area, data storage area and temporary management area, performance resources are fixedly allocated to each area, and idle performance resources are reserved in the resource pool for dynamic resource allocation. Each area can work independently. The data transmission order is from the task release area to the data storage area, then from the data storage area to the temporary management area, and finally from the temporary management area back to the task release area.

4. The method for optimizing the read / write performance of a storage unit according to claim 1, characterized in that: In step S2, a monitoring linkage is established with the receiving module of the storage unit to obtain the transmission data received by the receiving module; secondly, the stage length is set according to the historical data transmission volume and the performance resources allocated in the task release area, and then sampling is performed according to the stage length.

5. The method for optimizing the read / write performance of a storage unit according to claim 4, characterized in that: The steps for sampling based on stage length are as follows: S2.1 When the receiving module detects that a transmission data input has been generated, the transmission data is sampled in stages according to the set stage length, so as to obtain the stage data corresponding to the transmission data sampling. Each stage of data corresponds to the transmission data of its respective stage length; After the transmitted data was detected, phased sampling was performed in a timely manner. However, the transmitted data was not fully received during this process. S2.2 Perform data label analysis on the data of each stage to obtain the data labels corresponding to the data of each stage; S2.

3. Based on the data tags of the data at each stage, release transmission tasks to form a transmission task set; The transmission task includes stage data and the transmission data within the corresponding stage. The number of transmission tasks is rounded up to the nearest integer.

6. The method for optimizing the read / write performance of a storage unit according to claim 5, characterized in that: In step S3, the data storage area includes multiple sub-storage areas, and the specific steps are as follows: S3.

1. Generate a region profile based on the initial data of the sub-storage area and obtain the region profile corresponding to the sub-storage area; S3.2 Perform region adaptation analysis on the region profile of each sub-storage area and the data label corresponding to the transmission task to obtain the adaptation degree between each sub-storage area and the transmission task. Then select the sub-storage area with the highest adaptation degree to accept the transmission task and form the transmission association between the sub-storage area and the transmission task. S3.2.1 Set the adaptation threshold based on performance parameters; S3.2.2 Compare the fit between the transmission task and the sub-storage area with the fit threshold. If the fit is less than the fit threshold, cancel the transmission association between the sub-storage area and the transmission task, and create a new sub-storage area with the data tag of the transmission task. Then, associate the newly created sub-storage area with the transmission task. Conversely, if the fit is greater than the fit threshold, monitoring will continue.

7. The method for optimizing the read / write performance of a storage unit according to claim 6, characterized in that: In step S4, the temporary management area temporarily stores the data of untransmitted tasks. The specific steps are as follows: S4.1 The temporary management area extracts the transmitted data of the corresponding transmission task from the sub-storage area, performs individual adaptation analysis on the transmitted data and the regional profile of the corresponding sub-storage area, obtains the adaptation degree between the transmitted data and the sub-storage area, and compares it with the adaptation threshold. S4.2 is only performed when the adaptation degree of the comparison result is less than the adaptation threshold. S4.

2. Pause the transmission tasks corresponding to the transmitted data whose adaptability is less than the adaptability threshold, and gradually transfer the transmitted data until the transmitted data in the sub-storage area meets the adaptability threshold. When the transmitted data is transferred, it is restored to the untransmitted data of the transmission task, and the untransmitted data of the transmission task is transferred to the temporary management area; S4.3 Calculate the bytes of data already transmitted in the sub-storage area, and perform a second stage division on the untransmitted data of the same transmission task based on the bytes of the transmitted data to obtain the untransmitted data of the second stage division. Then, the task release area performs data tag analysis and task release on the untransmitted data of the second stage division.

8. The method for optimizing the read / write performance of a storage unit according to claim 1, characterized in that: In S5, corresponding idle performance resources are prepared according to the amount of stage data, and the task release area, data storage area and temporary management area are dynamically scheduled. Set a scheduling resource threshold. When the remaining available performance resources in any of the task release area, data storage area, and temporary management area are lower than the scheduling resource threshold, the corresponding idle performance resources will be used to supplement the performance resources.

9. A storage cell read / write performance optimization system, used to implement the storage cell read / write performance optimization method according to any one of claims 1-8, characterized in that: It includes a performance management module, a task publishing module, a data storage module, and a temporary management module; The performance management module is used to obtain the performance parameters of the storage unit, perform modular partitioning of the storage unit to obtain the task release area, data storage area and temporary management area, and allocate performance resources to each area according to the performance parameters, while reserving idle performance resources and dynamically allocating idle performance resources to the task release area, data storage area and temporary management area. The task publishing module includes a task publishing area, a receiving module for monitoring and storing the storage unit, a phase length for sampling the transmission data obtained by the receiving module in stages, and a phase publishing of transmission tasks based on the data tags corresponding to the phase data obtained by the phase sampling. The data storage module includes a data storage area. Based on the initial data in the data storage area, a corresponding regional profile is obtained. Based on the regional profile and data tags, a regional adaptation analysis is performed. Based on the analysis results, sub-regions and initial transfer data are generated until the matching degree meets the transmission task. Then, the transmission task at the corresponding stage is accepted. The temporary management module includes a temporary management area. It monitors the transmitted data corresponding to the transmission task in the data storage area, performs individual adaptation analysis on the transmitted data and the regional profile, stops and transfers the untransmitted data of the transmission task based on the analysis results, and updates the transmission task in the task publishing area based on the bytes of the transmitted data.

Images