Data storage method, electronic terminal and computer readable storage medium

Abstract

The invention provides a data storage method, an electronic terminal and a computer readable storage medium, and the method comprises the steps: dividing each piece of received image data into a plurality of data units through a current data node server corresponding to a current point location; receiving a data node list, and distributing the plurality of data units to the data node server in the data node list; according to the method, the similar image data of the same point location is merged and then compressed, so that the compression ratio can be improved, and the consumption of computing resources is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, in particular to a data storage method, an electronic terminal and a computer readable storage medium. BACKGROUND

[0002] In the field of cloud storage, there are great challenges in compression rate, compression speed and compression delay for mass picture compression storage. The compression storage of mass small pictures has problems such as low compression rate, high compression delay and large amount of metadata generated by compression.

[0003] Traditional compression methods are mostly optimized in the compression algorithm of the picture itself, and the optimization space of small picture compression efficiency is limited due to the compression algorithm itself. And the compression action generally selects the algorithm related optimization before the small picture is written into the cloud storage. Each small picture triggers a compression action, and the data entry is easy to cause a calculation resource bottleneck, while the data node data remains more, and the data utilization rate is low. When the small pictures are compressed alone, each compression will generate metadata of compressed data, there are a large amount of redundant data, resulting in a decrease in the final compression rate; and multiple compression, only a small amount of data is processed each time, more calculation resources are consumed. SUMMARY

[0004] The present application mainly provides a data storage method, an electronic terminal and a computer readable storage medium, which can improve the compression rate and reduce the consumption of calculation resources.

[0005] To solve the above technical problems, the first technical solution adopted by the present application is to provide a data storage method, which is applied to a data node server, and the method comprises: Dividing each image data received by a current data node server corresponding to a current point position into a plurality of data units; Receiving a data node list and distributing the plurality of data units to the data node servers in the data node list; Storing the distributed data units according to point positions.

[0006] In an embodiment, before receiving the data node list, the method comprises: Sending a request instruction to a metadata server based on the number of data units to request to obtain the data node list.

[0007] In an embodiment, the storing of the distributed data units according to point positions comprises: Storing the data units to the tail of a target physical file corresponding to the point position.

[0008] In an embodiment, after storing the data units to the tail of the target physical file corresponding to the point position, the method further comprises: Determine whether the size of the target physical file has reached the threshold; In response, the target physical file is compressed and stored.

[0009] In one embodiment, in response to no, it is determined whether the target physical file is in a writable state; If the response is no, the target physical file is compressed and stored.

[0010] In one embodiment, the method further includes: The resource availability rate of the data node server is reported, which is determined based on CPU utilization, memory availability, capacity availability and disk I / O availability.

[0011] To solve the above-mentioned technical problems, the third technical solution adopted by the present invention is: to provide a data storage method, the method being applied to a metadata server, the method comprising: Identify available data nodes; A list of data nodes is generated based on the available data nodes and sent to the current data node server so that the current data node server can allocate data units to the data node servers in the list of data nodes. The data unit is obtained by the current data node server after dividing each image data.

[0012] In one embodiment, determining available data nodes includes: The available data nodes are determined based on the resource availability of the data node servers; Among them, the resource availability of the data node servers in the data node list is greater than the threshold.

[0013] To solve the above-mentioned technical problems, the third technical solution adopted by the present invention is: to provide an electronic terminal, the electronic terminal including a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory, and the processor being used to execute program data to implement the steps in the method described above.

[0014] To solve the above-mentioned technical problems, the fourth technical solution adopted by the present invention is to provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps in the method described above.

[0015] The beneficial effects of this invention are as follows: Unlike the prior art, the data storage method provided in this application includes: dividing each received image data into multiple data units using the current data node server corresponding to the current point; receiving a data node list and allocating the multiple data units to the data node servers in the data node list; and merging and storing the allocated data units according to the point. This method can improve the compression rate and reduce the consumption of computing resources by merging similar image data of the same point and then compressing them. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the first embodiment of the data storage method of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the data unit merging and storage of the present invention; Figure 3 This is a flowchart illustrating the second embodiment of the data storage method of the present invention; Figure 4 This is a flowchart illustrating the third embodiment of the data storage method of the present invention; Figure 5 This is a flowchart illustrating the fourth embodiment of the data storage method of the present invention; Figure 6 This is a schematic diagram of the framework of an embodiment of the electronic terminal of the present invention; Figure 7 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of the present invention. Detailed Implementation

[0018] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0019] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0020] In this article, the term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "more" in this article means two or more objects.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0023] The data storage method provided in this application can be implemented by a server or terminal alone, or by a server and terminal working together. In some embodiments, the terminal or server can implement the data storage method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client that supports virtual scenes, such as a game APP; it can also be a mini-program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0024] It should be noted that the data storage method of this application is applied to a cloud storage system, which is a distributed data storage system composed of one or more metadata servers and data node servers. This application describes one metadata server and multiple data node servers. The data node server is a service node for storing actual file data. It can be an independent data storage device dedicated to storing data, or it can be a communication device with data storage function; the specific type is not limited. The metadata server is a dedicated server deployed with metadata management services. Its main responsibility is to store and manage metadata, rather than actual file data.

[0025] To enable those skilled in the art to better understand the technical solution of the present invention, a data storage method provided by the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Please see Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data storage method of the present invention. The data storage method of this embodiment is applied to a data node server and includes: Step S11: Use the current data node server corresponding to the current point to divide each received image data into multiple data units.

[0027] Specifically, once a fixed-point image acquisition device is stationary, its angle remains essentially unchanged, resulting in image data with largely similar backgrounds, differing only in moving objects. If these images are compressed individually, each image requires a separate compression dictionary (i.e., metadata), generating a large amount of compression metadata and reducing the final compression ratio. Furthermore, compressing individual images involves multiple compression steps, processing only a small amount of data from each image each time, consuming more computational resources.

[0028] Each fixed location corresponds to a data node server that stores the image data acquired by the image acquisition device at that location. This application uses the data node server (i.e., the current data node server) corresponding to a single location (i.e., the current location) as an example for explanation. The current data node server continuously receives a large amount of image data acquired by the image acquisition device at the current location. Each received image data is segmented into multiple data units. In one embodiment, the received image data is read into memory frame by frame using streaming media, and then each image data is segmented to obtain multiple data units.

[0029] In one embodiment, the data unit includes data pieces and check blocks. Specifically, data piece technology is used to divide each image data into multiple data pieces. The core purpose of data piece is to adapt to the redundancy protection requirements of EC (Erasure Coding) technology and improve the efficiency of distributed storage. EC (Erasure Coding) is a data redundancy protection technology that splits image data into multiple data pieces and generates check blocks. Through the combination of data pieces and check blocks, data recovery is achieved after the loss of some blocks. For example, the common EC rule "N+M" (such as 4+2) means that the image data is split into N data pieces, and then M check blocks are calculated and generated. Finally, N+M data units are used to store the image data together. Even if M blocks (whether data pieces or check blocks) are lost, the complete data can be recovered.

[0030] In one embodiment, the image data is evenly divided into N equal-length data slices according to the EC rule. For example, if the image data is 100k, splitting it according to the "5+2" EC rule will result in five 20k data slices. Then, two check blocks for verification are calculated based on these five data slices, ultimately resulting in 5+2 data units.

[0031] Step S12: Receive the data node list and assign the plurality of data units to the data node servers in the data node list.

[0032] After dividing the image data into multiple data units, the data node server receives a list of data nodes from the metadata server and assigns the multiple data units to the data node servers in the data node list.

[0033] In one specific embodiment, the data node server sends a request instruction to the metadata server based on the number of data units, requesting to obtain the data node list. Specifically, the SDK interface is invoked to send the request instruction to the data server. Continuing the above example, the number of data units is 7, including 5 data shards and 2 check blocks. Therefore, the data node server sends a request instruction to the metadata server, informing the metadata server that 7 data node servers are currently needed for data storage. The metadata server then returns the data node list to the data node server based on this information.

[0034] In one embodiment, the number of data node servers in the data node list returned by the metadata server is the same as the number of data units. In this case, the data node servers allocate the data units one-to-one to each data node server in the data node list.

[0035] In one embodiment, if the number of data node servers in the data node list returned by the metadata server is greater than the number of data units, the data node servers will randomly allocate the data units to some of the data node servers in the data node list. Alternatively, the data node servers will allocate the data units to data node servers in the data node list with high resource availability.

[0036] In another embodiment, if the number of data node servers in the data node list returned by the metadata server is less than the number of data units, then the data node server can allocate multiple data units to some of the data node servers in the data node list, or distribute the data units equally among the data node servers in the data node list, or allocate data units according to the resource availability of the data node servers in the data node list, such as allocating more data units to data node servers with high resource availability and allocating fewer data units to data node servers with low resource availability.

[0037] In one embodiment, the data storage method of this application further includes: reporting the resource availability rate of the data node server, wherein the resource availability rate is determined based on CPU utilization, memory availability, capacity availability, and disk I / O availability. Specifically, each data node server also needs to report its own resource availability rate.

[0038] In one embodiment, each data node server determines its resource availability based on its remaining CPU utilization X (100% minus total utilization), remaining available memory Y (100% minus total utilization), available capacity Z (remaining capacity as a percentage of total capacity), and disk I / O U availability (100% minus I / O load percentage).

[0039] Because different computing resources have varying impacts on compression, different weights are used for calculation. During compression, CPU is the most demanding resource, followed by memory and disk I / O. Considering the capacity load balancing of the cloud storage system, the available capacity weight is placed before disk I / O. Specifically, the CPU weight is, for example, 70%, the memory weight is, for example, 15%, the capacity weight is, for example, 10%, and the disk I / O weight is, for example, 5%. Using the above information, the resource availability rate of the data node server is calculated as C = X * 70% + Y * 15% + Z * 10% + U * 5%. After calculating its own resource availability rate, the data node server reports the resource availability rate to the metadata server.

[0040] In another embodiment, each data node server may only report information such as CPU utilization, memory availability, capacity availability, and disk I / O availability to the metadata server. The metadata server calculates the resource availability of each data node server based on the reported information.

[0041] In one embodiment, the metadata server further sorts the resource availability of each data node server from high to low, placing the top 70% of nodes in the candidate queue and the bottom 30% in the busy queue, awaiting the next sort. If the resource availability of a data node server increases, moving it from the bottom 30% to the top 70%, the data node server is moved from the busy queue to the candidate queue. Conversely, if the resource availability decreases, placing it in the bottom 30% after re-sorting, it is moved from the candidate queue to the busy queue. This method avoids having no available nodes, uses computing resources as the primary measure of load balancing, ensures full utilization of idle nodes, improves overall cluster resource utilization, and accelerates compression processing speed.

[0042] When the metadata server generates a list of data nodes, it can add the top 70% of the data node servers to the list.

[0043] Each data node server establishes a communication connection through the network, and the data node server at the corresponding location distributes data units to each data node server in the data node list through the network.

[0044] Step S13: The allocated data units are merged and stored according to their locations.

[0045] It should be noted that the list of data nodes returned by the metadata server generally includes the current data node server corresponding to the current point, that is, the current data node server will participate in the storage of the data units of the image data corresponding to the current point.

[0046] Combination Figure 2 Suppose that one image data point (point 1) is divided into 7 data units (A1-G1), where A1-E1 are data fragments, and F1 and G1 are check blocks. These 7 data units are then allocated to 7 data node servers (A2-G2). Assume that data node server A2 is the current data node server, and that the current data node server has created 3 physical files F1, F2, and F3. Physical file F1 stores the data units from point 1, physical file F2 stores the data units from point 2, and physical file F3 stores the data units from point 3.

[0047] For the current data node server A2, if it is allocated data unit A1 from image data of point 1 (the current node), then data unit A1 is stored in physical file F1. The next time image data is acquired and a corresponding data unit A1 is allocated, data unit A1 is also stored in physical file F1 corresponding to point 1. This merges the allocated data units corresponding to point 1 into physical file 1 for storage. For point 2, if its data unit A1 is allocated to data node server A2 (at this time, data node server B2 corresponds to point 2), then data node server A2 stores the allocated data unit A1 in physical file F2. The next time it is allocated data units for point 2, it also stores the data unit in physical file F2. This merges the allocated data units corresponding to point 2 into physical file 2 for storage.

[0048] In one embodiment, the data unit is stored at the end of the target physical file corresponding to the point. Specifically, for multiple image data acquired at point 1, assuming that data unit A1 obtained from image data 1 is stored in physical file F1, then data unit A1 obtained from image data 2 will be stored in physical file F1 (the target physical file corresponding to point 1) after data unit A1 of image data 1. Subsequently, after image data 3 is received, data unit A1 obtained from image data 3 will be stored in physical file F1 after data unit A1 of image data 2.

[0049] By using the above method, data units obtained from dividing image data at the same location are written into the same target physical file, which can merge data units at the same location into a large file.

[0050] In one embodiment, after storing the data unit at the end of the target physical file corresponding to the point, it is further determined whether the size of the target physical file reaches a threshold. In response, the target physical file is compressed and stored.

[0051] Specifically, the compression algorithm in this application is based on EC encoding technology. To adapt to EC encoding technology, a threshold is set for the size of the target physical file. In one embodiment, the threshold is assumed to be 256MB. The threshold depends on the mode of EC encoding technology; different modes result in different thresholds. In this embodiment, after storing the data units at the end of the target physical file corresponding to the specified point, and merging the data units at the same point into a larger file, it is further determined whether the size of the target physical file reaches the threshold. If so, it indicates that the target physical file has met the encoding conditions of EC encoding technology. At this point, EC encoding technology is used to compress and store the target physical file.

[0052] Since the size of the data units is consistent, the data node server in the data node list will trigger compression judgment logic after storing the data unit at the end of the target physical file corresponding to the point, that is, to determine whether the size of the target physical file has reached the threshold.

[0053] When compressing a target physical file, compression algorithms (such as LZMA and Zstd) can be used. These algorithms can utilize the continuous data stream of a large file to find repeated strings over a longer distance, thereby achieving acceleration at the binary level and obtaining a higher compression ratio and speed than a single image.

[0054] Furthermore, if the size of the target physical file does not reach the threshold, it is further determined whether the target physical file is in a writable state. If not, the target physical file is compressed and stored. Specifically, it is determined whether the target physical file is closed. If it is closed, it is in a non-writable state; if it is not closed, it is in a writable state.

[0055] If the target physical file is in a writable state, storage will continue during the next data storage unit.

[0056] Specifically, after each data node server receives a data unit, it appends the data unit to the end of the corresponding physical file, thus merging image fragments from the same location into a large file. Because a logical filename strongly related to the location name and storage time is created before the image data for that location is written, subsequent writes of data from the same location all use the same logical large file until the logical large file is full and closed, at which point a new physical file is switched. After switching to a new physical file, the writing logic remains unchanged, so fragments of similar images are always written to the corresponding physical file.

[0057] After the target physical file is compressed and stored, the original image data written in memory is cleaned up and deleted.

[0058] In one embodiment, combined with Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the data storage method of this application. This embodiment uses the compressed storage of a single image of point 1 as an example. After receiving the image data, the image data is segmented to obtain multiple (assuming 3) data units. These 3 data units are allocated to data node servers 1 to 3. Data node server 1 stores the allocated data units in physical file 1 (physical file 1 of data node server 1 is used to store the data units of point 1), data node server 2 stores the allocated data units in physical file 2 (physical file 2 of data node server 2 is used to store the data units of point 1), and data node server 3 stores the allocated data units in physical file 3 (physical file 3 of data node server 3 is used to store the data units of point 1). If the size of physical files 1 to 3 reaches a threshold, physical files 1 to 3 are compressed and stored as a whole, thus achieving data storage on disk.

[0059] In this embodiment, each data node server independently compresses the merged data units as a whole, which can make full use of the computing resources of each node in the cluster. The pressure of compression is distributed to the remaining nodes in the cluster. Since the data at the same location has higher similarity, the compression effect of duplicate data after merging is better, and a higher compression ratio can be obtained than for a single image. Since the size of the data fragments appended by each data node server is consistent (i.e., the size of each data unit is consistent), the data compression action will be triggered simultaneously to determine whether the physical file size has reached the threshold, which can reduce the compression time consumption, that is, reduce the compression latency.

[0060] The traditional compression process is as follows: receive an image - compress individual images - fragment the compressed data - store it in a physical file, thus achieving compressed disk storage. This traditional compression process compresses individual images and writes them to cloud storage, resulting in low resource availability, slow processing speed, and the compression ratio is affected by the compression metadata of numerous small images, leading to an even lower overall compression ratio.

[0061] Figure 3The compression process of this application is as follows: receiving image - data fragmentation - data unit - storage in physical file - whole-block compression, achieving compression and disk storage. This application's storage method fully utilizes the similarity of image data from the same location and the distributed nature of distributed storage files. It abandons the optimization approach of traditional compression algorithms, merging similar image data from the same location before compression, further compressing the file size and reducing the generation of metadata for small single-image compression. Utilizing multi-node resources of cloud storage, the merged data is compressed concurrently, significantly reducing compression latency and improving compression processing throughput compared to single-image compression at the data stream inlet.

[0062] See Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the data storage method of this application. The data storage method of this embodiment is applied to a metadata server, and includes: Step S41: Identify available data nodes.

[0063] In one embodiment, the metadata server can determine the available data nodes based on the resource availability of the data node servers.

[0064] In one embodiment, each data node server determines its resource availability based on its remaining CPU utilization X (100% minus total utilization), remaining available memory Y, available capacity Z, and disk IOU availability, and reports the resource availability to the metadata server. In another embodiment, each data node server reports its remaining CPU utilization X (100% minus total utilization), remaining available memory Y, available capacity Z, and disk IOU availability to the metadata server, which then calculates the resource availability based on these parameters.

[0065] The metadata server further sorts the resource availability of each data node server from highest to lowest, placing the top 70% of nodes in the candidate queue and the bottom 30% in the busy queue, awaiting the next sort. If the resource availability of a data node server increases, moving it from the bottom 30% to the top 70%, the data node server is moved from the busy queue to the candidate queue. Conversely, if the resource availability decreases, placing it in the bottom 30% after re-sorting, it is moved from the candidate queue to the busy queue. This method avoids having no available nodes, uses computing resources as the primary measure of load balancing, ensures full utilization of idle nodes, improves overall cluster resource utilization, and accelerates compression processing.

[0066] When the metadata server generates a list of data nodes, it can add the top 70% of the data node servers to the list. This means the resource availability of the data node servers in the list is greater than a threshold.

[0067] Step S42: Generate a list of data nodes based on the available data nodes and send it to the current data node server.

[0068] Specifically, a data node list is generated and sent to the current data node server, so that the current data node server allocates data units to the data node servers in the data node list; wherein, the data unit is obtained by the current data node server after dividing each image data.

[0069] This application's data storage method, through cross-image redundancy elimination, multi-node concurrent compression strategies, and a load balancing design prioritizing computational resources, achieves the goals of reducing compression overhead, lowering compression latency, and improving resource utilization, thereby accelerating compression speed while further reducing the compressed file size. Based on the inherent similarity of data captured from the same location, image fragments are merged into sufficiently large files for unified compression. Utilizing the continuous data stream of large files, cross-image repetition patterns and longer-distance repetitive strings can be identified, thus reducing compression overhead and the amount of metadata after compression. Combining the characteristics of distributed system file storage, the data compression process is shifted from the traditional number-stream entry compression method to concurrent compression at data storage nodes. Specifically, the traditional method compresses the entire image data on data node servers; this application's method fragments the image data and uses multiple data node servers for parallel compression, distributing the load across multiple nodes in the cluster, improving the overall compression throughput of the cluster, and reducing compression latency. When requesting data write nodes (i.e., determining data node servers), the metadata server weighs the available computational resources of each data node server, employing a load balancing strategy that prioritizes computational resources over capacity to fully balance cluster computational resources, improve resource utilization, and accelerate cluster compression speed.

[0070] See Figure 5 , Figure 5 This is a flowchart illustrating the fourth embodiment of the data storage method of this application. The data storage method of this embodiment is applied to a cloud storage system, which includes one metadata server and multiple data node servers.

[0071] This embodiment uses data node server 1 as an example. After receiving image data, data node server 1 segments the image data to obtain multiple (assuming 3) data units. It requests a data node list from the metadata server. The metadata server returns the data node list based on the resource availability of each data node server. Assuming the data node list includes data node servers 1 to 3, the 3 data units are allocated to data node servers 1 to 3 respectively. Data node server 1 stores the allocated data units in physical file 1 (physical file 1 of data node server 1 is used to store the data units of point 1), data node server 2 stores the allocated data units in physical file 2 (physical file 2 of data node server 2 is used to store the data units of point 1), and data node server 3 stores the allocated data units in physical file 3 (physical file 3 of data node server 3 is used to store the data units of point 1). If the size of physical files 1 to 3 reaches a threshold, physical files 1 to 3 are compressed and stored as a whole to achieve data persistence to disk.

[0072] Please see Figure 6 , Figure 6 This is a schematic diagram of the framework of an embodiment of the electronic terminal of the present invention. The electronic terminal 80 includes a memory 81 and a processor 82 coupled to each other. The processor 82 is used to execute program instructions stored in the memory 81 to implement the steps of any of the above-described data storage method embodiments. In a specific implementation scenario, the electronic terminal 80 may include, but is not limited to, a microcomputer or a server. In addition, the electronic terminal 80 may also include mobile devices such as laptops and tablets, which are not limited here.

[0073] Specifically, processor 82 controls itself and memory 81 to implement the steps of any of the above-described data storage and data retrieval method embodiments. Processor 82 may also be referred to as a CPU (Central Processing Unit). Processor 82 may be an integrated circuit chip with signal processing capabilities. Processor 82 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 82 may be implemented using integrated circuit chips.

[0074] Please see Figure 7 , Figure 7 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of the present invention. The computer-readable storage medium 90 stores program instructions 901 that can be executed by a processor. The program instructions 901 are used to implement the steps of any of the above-described data storage method embodiments.

[0075] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0076] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0078] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0079] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0080] The above are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A data storage method, characterized in that, The method is applied to a data node server, and the method includes: Each received image data is divided into multiple data units using the current data node server corresponding to the current location; Receive a list of data nodes and assign the plurality of data units to the data node servers in the list of data nodes; The allocated data units are merged and stored according to their locations.

2. The data storage method according to claim 1, characterized in that, Before receiving the list of data nodes, the following is included: Based on the number of data units, a request instruction is sent to the metadata server to request the list of data nodes.

3. The data storage method according to claim 2, characterized in that, The step of merging and storing the allocated data units according to their locations includes: The data unit is stored at the end of the target physical file corresponding to the point.

4. The data storage method according to claim 3, characterized in that, After storing the data unit at the end of the target physical file corresponding to the point, the method further includes: Determine whether the size of the target physical file has reached the threshold; In response, the target physical file is compressed and stored.

5. The data storage method according to claim 4, characterized in that, In response to no, determine whether the target physical file is in a writable state; If the response is no, the target physical file is compressed and stored.

6. The data storage method according to any one of claims 1 to 5, characterized in that, The method further includes: The resource availability rate of the data node server is reported, which is determined based on CPU utilization, memory availability, capacity availability and disk I / O availability.

7. A data storage method, characterized in that, The method is applied to a metadata server, and the method includes: Identify available data nodes; A list of data nodes is generated based on the available data nodes and sent to the current data node server so that the current data node server can allocate data units to the data node servers in the list of data nodes. The data unit is obtained by the current data node server after dividing each image data.

8. The data storage method according to claim 7, characterized in that, Identify available data nodes, including: The available data nodes are determined based on the resource availability of the data node servers; Among them, the resource availability of the data node servers in the data node list is greater than the threshold.

9. An electronic terminal, characterized in that, The electronic terminal includes a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory, and the processor being used to execute program data to implement the steps in the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 8.

Images