Data storage load balancing method, apparatus, device, and storage medium

Abstract

The application discloses a data storage load balancing method, device and equipment and a storage medium, and the method comprises the following steps: receiving all file data to be stored; mapping the file data to a two-dimensional coordinate system with a preset size after vectorization, and confirming the two-dimensional coordinate points corresponding to each file data and the size of each file data; based on the coordinate descent method, the X axis and the Y axis of the two-dimensional coordinate system are divided into multiple regions by using a dynamic adjustment mode, and during the dynamic adjustment process, the maximum value of the total size of the file data in all regions continuously decreases until the maximum value meets a preset requirement, and the optimal division of the X axis and the Y axis is obtained; and the file data is stored in the disk space after being divided according to the optimal division. After the file data is mapped to a two-dimensional space, the file data is divided by combining the dynamic programming and the coordinate descent idea, so that the file data is stored in the disk in a balanced manner.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data storage load balancing method, apparatus, device, and storage medium. Background Technology

[0002] With the popularization and development of the internet, people can more easily create, share, and store data. However, the proliferation of social media, sensor technology, mobile devices, and cloud computing has led to an explosive growth in data, with the volume increasing exponentially. Compared to data warehouses and data lakes, lake warehouses are better able to meet users' diverse needs for big data platforms. A lake warehouse is a data management and analysis architecture designed to bridge the gap between traditional data lakes and data warehouses, integrating their advantages for better data management and analysis. Simultaneously, the lake warehouse architecture emphasizes data quality and governance, ensuring that data undergoes cleaning and verification before entering the lake warehouse, thereby improving data reliability and availability.

[0003] However, in a lake warehouse, data originates from various business application systems, and these systems contain diverse data formats, including structured, unstructured, and semi-structured data, as well as various data types such as images, text, CSV, HTML, and XML. After file data enters the lake warehouse, metadata is extracted. This metadata allows for file classification based on attributes, names, text content, and creation times, grouping files with similar characteristics together. Deep learning algorithms can be used to train models for automatic file classification. However, similar files often have significant content overlap, potentially leading to uneven disk storage. A trade-off between space utilization and balanced file distribution is necessary. Current technologies focus on text similarity classification, whether classifying based on title and keyword combinations in metadata or improving clustering algorithms by extracting text content. However, none have achieved a balanced distribution of file data. Uneven file data distribution on disk means some areas may be frequently accessed while others are rarely accessed, hindering subsequent file access. Imbalanced access to file data can lead to performance degradation. Currently, some clustering algorithms, such as hierarchical clustering, allow adjusting the size or number of clusters by pruning or merging branches within the tree structure to balance the cluster size. However, these methods rely on the original parameters and settings; inappropriate selection can have the opposite effect and negatively impact clustering quality. Summary of the Invention

[0004] In view of this, this application provides a data storage load balancing method, apparatus, device and storage medium to solve the problem of uneven storage on disk caused by existing data storage methods.

[0005] To address the aforementioned technical problems, this application provides a data storage load balancing method, characterized by comprising: receiving all file data to be stored; vectorizing the file data and mapping it to a two-dimensional coordinate system of a preset size, and confirming the two-dimensional coordinate point corresponding to each file data and the size of each file data; based on the coordinate descent method, dynamically adjusting the X-axis and Y-axis of the two-dimensional coordinate system into multiple regions, and during the dynamic adjustment process, the maximum value of the sum of the file data sizes in all regions continuously decreases until the maximum value meets the preset requirements, thereby obtaining the optimal division of the X-axis and Y-axis; and storing the file data in disk space after being divided according to the optimal division.

[0006] As a further improvement of this application, based on the coordinate descent method, the X-axis and Y-axis in the two-dimensional coordinate system are divided into multiple regions using a dynamic adjustment method. During the dynamic adjustment process, the maximum value of the sum of file data in all regions continuously decreases until the maximum value meets a preset requirement, thus obtaining the optimal division of the X-axis and Y-axis. This includes: dividing the Y-axis in the two-dimensional coordinate system into a preset number of regions and keeping them unchanged; repeatedly performing the operation of adjusting the division of the X-axis, and during the adjustment process, the maximum value of the sum of file data in all regions continuously decreases until the maximum value meets a preset requirement, thus obtaining the optimal division of the X-axis; while maintaining the optimal division of the X-axis, repeatedly performing the operation of adjusting the division of the Y-axis, and during the adjustment process, the maximum value of the sum of file data in all regions continuously decreases until the maximum value meets a preset requirement, thus obtaining the optimal division of the Y-axis.

[0007] As a further improvement to this application, the process of adjusting the X-axis or Y-axis division is expressed as follows:

[0008]

[0009]

[0010] Where f(i,k) represents the process of dividing the X-axis or Y-axis into k regions starting from index i, such that the maximum sum of the file data sizes in all the divided regions continuously decreases until a preset requirement is met, where S1, S2, ..., S... k The value represents the total size of the file data divided into each region, f(i+m,k-1) represents the result of the previous partition, and W represents the total size of the file data divided into each region. p Δ represents the total size of the file data. tThis represents the smallest scale division of the X-axis or Y-axis, and P represents the two-dimensional coordinate point corresponding to each region.

[0011] As a further improvement of this application, the file data is vectorized and then mapped to a two-dimensional coordinate system of a preset size, including: extracting metadata information representing the features of the file data from the file data; constructing feature vectors or feature sets based on the metadata information; clustering all feature vectors or feature sets to obtain at least one cluster, each cluster including at least one feature vector or feature set; and performing dimensionality reduction operations on the feature vectors or feature sets to map the feature vectors or feature sets to a two-dimensional coordinate system, generating two-dimensional coordinate points corresponding to each file data.

[0012] As a further improvement to this application, the metadata information includes at least one of the following: file name, file size, file type, and file creation time.

[0013] As a further improvement to this application, after generating the two-dimensional coordinate points corresponding to each file data, the method further includes: mapping the two-dimensional coordinate points onto a two-dimensional coordinate system and then displaying the output.

[0014] As a further improvement to this application, after extracting metadata information that characterizes the features of the file data from the file data, the method further includes: preprocessing the metadata information.

[0015] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a data storage load balancing device, comprising: a receiving module for receiving all file data to be stored; a mapping module for vectorizing the file data and mapping it to a two-dimensional coordinate system of a preset size, and confirming the two-dimensional coordinate point corresponding to each file data and the size of each file data; an optimization module for dividing the X-axis and Y-axis of the two-dimensional coordinate system into multiple regions based on the coordinate descent method and using a dynamic adjustment method, wherein during the dynamic adjustment process, the maximum value of the sum of the file data sizes of all regions continuously decreases until the maximum value meets the preset requirements, thereby obtaining the optimal division of the X-axis and Y-axis; and a classification storage module for storing the file data in disk space after dividing it according to the optimal division.

[0016] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a computer device, the computer device including a processor and a memory coupled to the processor, the memory storing program instructions, and when the program instructions are executed by the processor, causing the processor to perform the steps of the data storage load balancing method as described above.

[0017] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a storage medium storing program instructions capable of implementing any of the above-mentioned data storage load balancing methods.

[0018] The beneficial effects of this application are as follows: The data storage load balancing method of this application maps the received file data to a two-dimensional space. Within this two-dimensional space, it combines dynamic programming and coordinate descent techniques to partition the file data, obtaining the optimal partitioning result. Based on this optimal partitioning result, the file data is then partitioned and stored on the disk, ensuring that the file data is evenly distributed across the disk space. Furthermore, compared to methods based on text content for storage, the two-dimensional coordinate axis partitioning proposed in this invention uses dynamic programming, transforming the original X and Y axis partitioning problem into a subproblem of partitioning each small interval. This eliminates the need to scan the text content, significantly saving time. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a data storage load balancing method according to an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of the functional modules of the data storage load balancing device according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the structure of the storage medium according to an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0024] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0025] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0026] Figure 1 This is a flowchart illustrating the data storage load balancing method according to an embodiment of the present invention. It should be noted that if substantially the same result is achieved, the method of the present invention does not necessarily require further clarification. Figure 2 The illustrated process sequence is limited. For example... Figure 1 As shown, the data storage load balancing method includes the following steps:

[0027] Step S101: Receive all file data to be stored.

[0028] Step S102: After vectorizing the file data, map it to a two-dimensional coordinate system of a preset size, and confirm the two-dimensional coordinate point corresponding to each file data and the size of each file data.

[0029] Specifically, after receiving the file data to be stored, in order to facilitate the analysis of the file data, this embodiment vectorizes the file data according to its characteristics, and then maps the vectorized file data into a two-dimensional space for display in a two-dimensional coordinate system.

[0030] Furthermore, this embodiment can be extended to a multi-dimensional space. In a multi-dimensional space, in addition to considering the file size feature, other features such as file creation time and file name can also be considered. By comprehensively managing file data based on these features, a balanced distribution and efficient storage of data within the lake warehouse can be achieved. Therefore, the step S102, which involves vectorizing the file data and mapping it to a two-dimensional coordinate system of a preset size, specifically includes:

[0031] 1. Extract metadata information that characterizes the features of the file data from the file data.

[0032] Specifically, by extracting the file's metadata, we can obtain the file's basic characteristics, including file name, file size, and file type, which facilitates subsequent file management, classification, and organization.

[0033] Furthermore, after extracting metadata information that characterizes the features of the file data from the file data, the process also includes preprocessing the metadata information.

[0034] Specifically, after extracting metadata information from file data, it is necessary to preprocess the extracted metadata information to remove null values ​​and obvious deviation values.

[0035] Furthermore, the metadata information includes at least one of the following: file name, file size, file type, and file creation time.

[0036] Specifically, by combining data such as file name, file size, file type, and file creation time, the file data is expanded to a multi-dimensional space, providing more detailed feature information for subsequent file data partitioning, making the subsequent file data partitioning results more reasonable.

[0037] 2. Construct feature vectors or feature sets based on metadata information.

[0038] 3. Cluster all feature vectors or feature sets to obtain at least one cluster, and each cluster includes at least one feature vector or feature set.

[0039] Specifically, by using a pre-defined clustering method, such as K-means clustering, feature vectors or feature sets are clustered, thereby grouping file data into different clusters based on the similarity of metadata information, and classifying files with similar metadata features into the same cluster.

[0040] 4. Perform dimensionality reduction on the feature vectors or feature sets to map the feature vectors or feature sets to a two-dimensional coordinate system, generating two-dimensional coordinate points corresponding to each file data.

[0041] Specifically, after clustering, the high-dimensional metadata information is reduced in dimensionality and mapped to a lower-dimensional space for more efficient processing.

[0042] Furthermore, after generating the two-dimensional coordinate points corresponding to each file data, the process also includes mapping the two-dimensional coordinate points onto a two-dimensional coordinate system and then displaying the output.

[0043] Specifically, by mapping two-dimensional coordinate points onto a two-dimensional coordinate system and displaying the output, all file data becomes visible in two-dimensional space, making it easy to observe the distribution of each cluster obtained through clustering in two-dimensional space.

[0044] Step S103: Based on the coordinate descent method, the X-axis and Y-axis in the two-dimensional coordinate system are divided into multiple regions by dynamic adjustment. During the dynamic adjustment process, the maximum value of the sum of the file data in all regions continues to decrease until the maximum value meets the preset requirements, thus obtaining the optimal division of the X-axis and Y-axis.

[0045] Specifically, after mapping file data to a two-dimensional space, this embodiment combines dynamic programming and coordinate descent to partition this two-dimensional space. Each two-dimensional coordinate point in the two-dimensional space represents a file data point. After the file data is mapped to the two-dimensional coordinate system, the question is how to allocate the file data to the disk to achieve a load-balanced distribution, i.e., how to partition the X and Y axes of the two-dimensional coordinate system so that the total size of the file data in each region is uniform. The positional relationship between the X and Y axes represents the similarity between file data points in the higher-dimensional space. In this embodiment, based on the idea of ​​coordinate descent, dynamic programming is used to continuously adjust the allocation of the X and Y axes, so that the maximum value of the total size of the file data in all regions after partitioning continuously decreases until the maximum value of the total size of the file data in all regions meets the preset requirements. The dynamic programming idea is to decompose the original problem of partitioning the X and Y axes into several sub-problems, i.e., given the partitioning of the X axis, find the optimal partitioning of the Y axis; or given the partitioning of the Y axis, find the optimal partitioning of the X axis.

[0046] Furthermore, step S103 specifically includes:

[0047] 1. Divide the Y-axis in the two-dimensional coordinate system into a preset number of regions and keep them unchanged.

[0048] 2. Repeatedly perform the operation of adjusting the division of the X-axis. During the adjustment process, the maximum value of the sum of the file data in all regions continues to decrease until the maximum value meets the preset requirements, thus obtaining the optimal division of the X-axis.

[0049] 3. While maintaining the optimal division of the X-axis, repeatedly perform the operation of adjusting the division of the Y-axis. During the adjustment process, the maximum value of the total size of the file data in all regions continuously decreases until the maximum value meets the preset requirements, obtaining the optimal division of the Y-axis.

[0050] It should be noted that in this embodiment, the X-axis can also be divided into a preset number of regions and kept unchanged first. First, divide the Y-axis. After the division of the Y-axis is completed, then divide the X-axis.

[0051] In this embodiment, the adjustment process of dividing the X-axis or Y-axis is expressed as:

[0052]

[0053]

[0054] Among them, f(i,k) represents that after dividing the X-axis or Y-axis into k regions starting from index i, the maximum value in the total size of the file data in all divided regions continuously decreases until it meets the preset requirements. S1, S2,..., S k represents the total size of the file data in each divided region, f(i + m,k - 1) represents the previous division result, and W p represents the total size of the file data, and Δ t represents the minimum scale of the X-axis or Y-axis, and P represents the two-dimensional coordinate points corresponding to each region.

[0055] Specifically, assume that N file data are mapped to the two-dimensional coordinate system. Each file data corresponds to a two-dimensional coordinate point and the size of the file data, that is, (x, y, w), where (x, y) represents the coordinates of the two-dimensional coordinate point and w represents the size of the file data. Then divide the two-dimensional coordinates. Assume that the size of the two-dimensional coordinate system is L×L, and the X-axis and Y-axis are each divided into k segments. The minimum scale Δx(Δy) of the X-axis and Y-axis is L / 100k. First step, evenly divide the Y-axis into k segments, and the length of each segment is L / k = 1ooΔy, which is S y . The division S x of the X-axis is initialized to a null value. Second step, divide the X-axis into m lengths starting from index i(0 ≤ i < L), and compare the total size of the regions S1, S2,...S k divided within this interval. Take the part with the largest total size among them and compare it with the remaining f(i + m,k - 1) regions. Through iteration, the maximum value of the total size continuously decreases until the maximum value of the total size no longer decreases or the amount of decrease is lower than the preset threshold. Third step, sequentially deduce the above steps according to the above formula to obtain the optimal X-axis S xThe fourth step, based on the above three steps that have determined the X-axis division, is to use the idea from the second step to divide the Y-axis. y The partitioning process involves dividing the Y-axis into m segments starting from index i, and then comparing the resulting regions S1, S2, ... S within this partition. k The sum of the values ​​is taken, and the largest sum is compared with the remaining regions f(i+m,k-1). Through iteration, the maximum sum is continuously reduced until the maximum sum stops decreasing or the reduction is below a preset threshold. Finally, the optimal partition S of the X-axis and Y-axis is obtained. x ∪S y .

[0056] In this embodiment, a dynamic programming method is used to partition the X and Y axes, ultimately achieving data storage load balancing. The distribution of file data on disk space requires consideration of many factors, including ease of subsequent access and a rational overall space distribution. This embodiment achieves the effect of evenly distributing file data across disk space, resulting in global disk space optimization. Consequently, subsequent access and querying of file data in the lake repository will be more convenient, and its efficiency will be greatly improved.

[0057] Step S104: Divide the file data according to the optimal partition and store it in the disk space.

[0058] Specifically, after obtaining the optimal partition of file data in two-dimensional space, the file data is partitioned according to this optimal partitioning method, so that the partitioned file data can be evenly distributed to the disk space, achieving overall disk load balancing and global optimization.

[0059] The data storage load balancing method of this invention maps the received file data to a two-dimensional space. Within this space, it uses dynamic programming and coordinate descent to partition the file data, obtaining an optimal partitioning result. Based on this optimal partitioning, the file data is then divided and stored on disk, ensuring that the file data is evenly distributed across the disk space. Furthermore, compared to storing data based on text content, the two-dimensional coordinate axis partitioning proposed in this invention uses dynamic programming, transforming the original X and Y axis partitioning problem into a subproblem of partitioning each small interval. This eliminates the need to scan the text content, significantly saving time.

[0060] Figure 2 This is a functional module diagram of the data storage load balancing device according to an embodiment of the present invention. Figure 2 As shown, the data storage load balancing device 20 includes: a receiving module 21, a mapping module 22, an optimization module 23, and a classified storage module 24.

[0061] Receiver module 21 is used to receive all file data to be stored;

[0062] The mapping module 22 is used to vectorize the file data and map it to a two-dimensional coordinate system of a preset size, and to confirm the two-dimensional coordinate point corresponding to each file data and the size of each file data.

[0063] Optimization module 23 is used to divide the X-axis and Y-axis in the two-dimensional coordinate system into multiple regions based on the coordinate descent method and by using dynamic adjustment. During the dynamic adjustment process, the maximum value of the sum of the file data in all regions continues to decrease until the maximum value meets the preset requirements, so as to obtain the optimal division of the X-axis and Y-axis.

[0064] The classification storage module 24 is used to divide file data into partitions according to the optimal partition and store them in the disk space.

[0065] Optionally, the optimization module 23 performs an operation based on coordinate descent, using dynamic adjustment to divide the X-axis and Y-axis of the two-dimensional coordinate system into multiple regions. During the dynamic adjustment, the maximum value of the sum of file data in all regions continuously decreases until the maximum value meets a preset requirement, thus obtaining the optimal division of the X-axis and Y-axis. Specifically, this includes: dividing the Y-axis of the two-dimensional coordinate system into a preset number of regions and keeping them unchanged; repeatedly performing the operation of adjusting the division of the X-axis, and during the adjustment, the maximum value of the sum of file data in all regions continuously decreases until the maximum value meets a preset requirement, thus obtaining the optimal division of the X-axis; while maintaining the optimal division of the X-axis, repeatedly performing the operation of adjusting the division of the Y-axis, and during the adjustment, the maximum value of the sum of file data in all regions continuously decreases until the maximum value meets a preset requirement, thus obtaining the optimal division of the Y-axis.

[0066] Optionally, the process of adjusting the X-axis or Y-axis division is expressed as follows:

[0067]

[0068]

[0069] Where f(i,k) represents the process of dividing the X-axis or Y-axis into k regions starting from index i, such that the maximum sum of the file data sizes in all the divided regions continuously decreases until a preset requirement is met, where S1, S2, ..., S... k The value represents the total size of the file data divided into each region, f(i+m,k-1) represents the result of the previous partition, and W represents the total size of the file data divided into each region. p Δ represents the total size of the file data. t This represents the smallest scale division of the X-axis or Y-axis, and P represents the two-dimensional coordinate point corresponding to each region.

[0070] Optionally, the mapping module 22 performs the operation of vectorizing the file data and mapping it to a two-dimensional coordinate system of a preset size. Specifically, this includes: extracting metadata information that characterizes the features of the file data from the file data; constructing feature vectors or feature sets based on the metadata information; clustering all feature vectors or feature sets to obtain at least one cluster, each cluster including at least one feature vector or feature set; and performing dimensionality reduction on the feature vectors or feature sets to map the feature vectors or feature sets to a two-dimensional coordinate system, generating two-dimensional coordinate points corresponding to each file data.

[0071] Optionally, the metadata information includes at least one of the following: file name, file size, file type, and file creation time.

[0072] Optionally, after the mapping module 22 performs the operation of generating two-dimensional coordinate points corresponding to each file data, it is also used to: map the two-dimensional coordinate points onto a two-dimensional coordinate system and then display the output.

[0073] Optionally, after performing the operation of extracting metadata information representing the characteristics of the file data from the file data, the mapping module 22 is further used to: preprocess the metadata information.

[0074] For further details regarding the implementation techniques of each module in the data storage load balancing device of the above embodiments, please refer to the description in the data storage load balancing method of the above embodiments, which will not be repeated here.

[0075] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0076] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Figure 3 As shown, the computer device 30 includes a processor 31 and a memory 32 coupled to the processor 31. The memory 32 stores program instructions. When the program instructions are executed by the processor 31, the processor 31 performs the data storage load balancing method steps described in any of the above embodiments.

[0077] The processor 31 can also be referred to as a CPU (Central Processing Unit). The processor 31 may be an integrated circuit chip with signal processing capabilities. The processor 31 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0078] See Figure 4 , Figure 4 This is a schematic diagram of the structure of a storage medium according to an embodiment of the present invention. The storage medium of this embodiment stores program instructions 41 capable of implementing the above-described data storage load balancing method. These program instructions 41 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or computer devices such as computers, servers, mobile phones, and tablets.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed computer devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0080] Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data storage load balancing method, characterized in that, It includes: Receive all file data to be stored; The file data is vectorized and mapped to a two-dimensional coordinate system of a preset size, and the two-dimensional coordinate point corresponding to each file data and the size of each file data are confirmed; Based on the coordinate descent method, the X-axis and Y-axis in the two-dimensional coordinate system are divided into multiple regions by dynamic adjustment. During the dynamic adjustment process, the maximum value of the sum of file data in all regions continues to decrease until the maximum value meets the preset requirements, thus obtaining the optimal division of the X-axis and Y-axis. The file data is divided and stored in disk space according to the optimal partition; wherein: The step of vectorizing the file data and mapping it to a two-dimensional coordinate system of a preset size includes: Extract metadata information that characterizes the features of the file data from the file data; Construct feature vectors or feature sets based on the aforementioned metadata information; Cluster all feature vectors or feature sets to obtain at least one cluster, and each cluster includes at least one feature vector or feature set; The feature vector or feature set is subjected to dimensionality reduction to map the feature vector or feature set to the two-dimensional coordinate system, generating two-dimensional coordinate points corresponding to each file data.

2. The data storage load balancing method according to claim 1, characterized in that, The coordinate descent method dynamically divides the X and Y axes of the two-dimensional coordinate system into multiple regions. During the dynamic adjustment process, the maximum sum of file data sizes in all regions continuously decreases until the maximum sum meets a preset requirement, thus obtaining the optimal division of the X and Y axes. This includes: The Y-axis in the two-dimensional coordinate system is divided into a predetermined number of regions and kept unchanged. The operation of adjusting the division of the X-axis is performed repeatedly, and during the adjustment process, the maximum value of the sum of the file data of all regions is continuously reduced until the maximum value meets the preset requirements, so as to obtain the optimal division of the X-axis. While maintaining the optimal division of the X-axis, the operation of adjusting the division of the Y-axis is performed cyclically. During the adjustment process, the maximum value of the sum of the file data in all regions is continuously reduced until the maximum value meets the preset requirements, thus obtaining the optimal division of the Y-axis.

3. The data storage load balancing method according to claim 2, characterized in that, The process of adjusting the X-axis or Y-axis division is represented as follows: ； ； in, Indicates from index Start dividing the X-axis or Y-axis After dividing the file into regions, the maximum value of the sum of the file data sizes in all regions is continuously reduced until a preset requirement is met. This represents the total size of the file data in each partitioned region. This indicates the result of the previous partitioning. This represents the total size of the file data. Represents the smallest scale division of the X-axis or Y-axis. This represents the two-dimensional coordinates of each region; m represents the number of divisions. Indicates from index Start by dividing the coordinate system into m lengths; L represents the size of the two-dimensional coordinate system.

4. The data storage load balancing method according to claim 3, characterized in that, The metadata information includes at least one of the following: file name, file size, file type, and file creation time.

5. The data storage load balancing method according to claim 4, characterized in that, After generating the two-dimensional coordinate points corresponding to each file data, the process further includes: The two-dimensional coordinate points are mapped onto the two-dimensional coordinate system and then displayed.

6. The data storage load balancing method according to claim 5, characterized in that, After extracting metadata information characterizing the features of the file data from the file data, the method further includes: The metadata information is preprocessed.

7. A data storage load balancing device utilizing the data storage load balancing method of claim 1, characterized in that, It includes: The receiving module is used to receive all file data to be stored; The mapping module is used to vectorize the file data and map it to a two-dimensional coordinate system of a preset size, and to determine the two-dimensional coordinate point corresponding to each file data and the size of each file data. The optimization module is used to divide the X-axis and Y-axis in the two-dimensional coordinate system into multiple regions based on the coordinate descent method and by using dynamic adjustment. During the dynamic adjustment process, the maximum value of the sum of the file data in all regions is continuously reduced until the maximum value meets the preset requirements, so as to obtain the optimal division of the X-axis and Y-axis. The classification storage module is used to divide the file data according to the optimal partition and store it in the disk space.

8. A computer device, characterized in that, The computer device includes a processor and a memory coupled to the processor, the memory storing program instructions that, when executed by the processor, cause the processor to perform the steps of the data storage load balancing method as described in any one of claims 1-6.

9. A storage medium, characterized in that, The system stores program instructions capable of implementing the data storage load balancing method as described in any one of claims 1-6.

Images