A data storage method, a cloud storage system, and a computer readable storage medium

Abstract

The application provides a data storage method, a cloud storage system and a computer readable storage medium. The data storage method is applied to a storage node in a cloud storage system. The data storage method comprises the following steps: accessing video data of a front-end device; acquiring a system network load of the storage node; and transmitting the video data to a disk of the storage node through local loopback transmission and / or network transmission based on the system network load. In this way, the cloud storage system utilizes the characteristics of high bandwidth and high transmission efficiency of the loopback network card, reduces the real network load, improves the network resource utilization, and improves the overall system performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data cloud storage, in particular to a data storage method, a cloud storage system and a computer readable storage medium. BACKGROUND

[0002] Video data generally has the characteristics of large data volume and continuous data, so the transmission of video data is often limited by network resources. The network resource utilization capability of a video cloud storage system is an important performance indicator of the storage system, which can directly reflect the upper limit of the accessible data volume of the system. Under the same conditions, a cloud storage system with high network resource utilization rate can have more business volume. With the continuous development of information technology and society, the video data volume in the field of security and other fields grows exponentially. While the data computing capability of the server hardware layer has been greatly improved, the network resources often show a bottleneck.

[0003] In addition, the current video cloud storage system also has the problem that the large amount of video data and limited network resources result in poor performance of the video cloud storage system and low network transmission efficiency. SUMMARY

[0004] The present application provides a data storage method, a cloud storage system and a computer readable storage medium.

[0005] The present application provides a data storage method, which is applied to a storage node in a cloud storage system. The data storage method comprises:

[0006] accessing video data of a front-end device;

[0007] obtaining a system network load of the storage node;

[0008] storing the video data to a disk of the storage node through local loop transmission and / or network transmission based on the system network load.

[0009] The method comprises:

[0010] when the system network load is greater than a first preset threshold, storing the video data to the disk of the storage node through the local loop transmission.

[0011] The method comprises:

[0012] when the system network load is less than a second preset threshold, storing the video data to the disk of the storage node through the network transmission; wherein the second preset threshold is less than the first preset threshold;

[0013] when the system network load is greater than the second preset threshold and less than the first preset threshold, storing part of the video data to the disk of the storage node through the network transmission according to a proportional relationship between the system network load and the preset threshold, and storing the rest of the video data to the disk of the storage node through the local loop transmission.

[0014] wherein the proportional relationship between the system network load and the preset threshold is a ratio relationship between the system network load and a sum of the first preset threshold and the second preset threshold.

[0015] wherein after the video data of the access front-end device, the data storage method further comprises:

[0016] obtaining a number of storage nodes of the cloud storage system;

[0017] when the number of storage nodes is 1, storing the video data to the disk of the storage node through the local loop transmission.

[0018] The application also provides another data storage method, which is applied to a cloud storage system, wherein the cloud storage system comprises a metadata management server and a storage node; the data storage method comprises:

[0019] the metadata management server obtains load information of the storage node, and generates a data writing mode according to the load information;

[0020] the storage node obtains the data writing mode distributed by the metadata management server, and writes video data of a front-end device to a disk according to the data writing mode.

[0021] wherein the load information comprises an average load of system resources of all storage nodes in the cloud storage system;

[0022] generating the data writing mode according to the load information comprises:

[0023] when the average load of system resources is greater than a preset load threshold, setting the data writing mode as the local loop transmission.

[0024] wherein the load information comprises a number of disks of the storage node and an average load of the disks;

[0025] generating the data writing mode according to the load information comprises:

[0026] write the data into the disk in a local loop transmission mode when the number of the disks is greater than a storage threshold of video data of the front-end device, or an average load of the disks is less than a preset disk threshold.

[0027] The storage threshold of the video data is determined by the split effective data block and the check data block.

[0028] The application further provides a cloud storage system, comprising a processor and a memory, wherein the memory stores program data, and the processor is configured to execute the program data to implement the data storage method as described above.

[0029] The application further provides a computer readable storage medium for storing program data, wherein the program data, when executed by a processor, is configured to implement the data storage method as described above.

[0030] The application has the beneficial effect that the storage nodes in the cloud storage system access video data of a front-end device, acquire system network load of the storage nodes, and store the video data into the disks of the storage nodes through local loop transmission and / or network transmission based on the system network load. In this way, the cloud storage system utilizes the characteristics of high bandwidth and high transmission efficiency of the loopback card, reduces real network load, improves network resource utilization, and improves overall system performance. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings. Among them:

[0032] Figure 1 is a system structure schematic diagram of an embodiment of the video cloud storage system provided by the present application;

[0033] Figure 2 is a schematic diagram of a data transmission mode of a conventional video cloud storage system;

[0034] Figure 3 is a schematic diagram of a data transmission mode of a video cloud storage system provided by the present application;

[0035] Figure 4 is a flowchart of an embodiment of the data transmission method provided by the present application;

[0036] Figure 5 is a flowchart of an access data flow conversion strategy provided by the present application;

[0037] Figure 6 This is a flowchart illustrating another embodiment of the data transmission method provided in this application;

[0038] Figure 7 This is a flowchart illustrating the cloud storage management service allocation strategy provided in this application;

[0039] Figure 8 This is a schematic diagram of the structure of an embodiment of the cloud storage system provided in this application;

[0040] Figure 9 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0042] To address the challenges of large video data volumes and limited network resources in video cloud storage systems, this application proposes a solution based on a loopback network interface card (NIC) to conserve network resources. Through algorithm optimization, the solution leverages the high transmission efficiency and low network resource consumption of the loopback NIC to further optimize network transmission efficiency and significantly improve the performance of the video cloud storage system.

[0043] The local loopback network card is a virtual network card simulated by software to handle network communication required by local software. On the one hand, the local loopback network card can work like a real network card, and since data packets are sent and received on a simulated terminal in actual operation, it has the characteristics of rapid network message processing and not actually following a network route; on the other hand, as a virtual network card, the local loopback network card does not experience disconnection, thus possessing high reliability.

[0044] Please refer to the details. Figure 1 , Figure 1 This is a schematic diagram of the system structure of an embodiment of the video cloud storage system provided in this application.

[0045] like Figure 1 As shown, the video cloud storage system of this application consists of two MDS (Management Data System) servers and one DN (Data Node), with an SDK (Software Development Kit) client serving as the entry point for user data requests.

[0046] Specifically, the metadata management server (MDS) is installed with access management services, storage management services, cloud storage management services and other management software, and makes load balancing decisions according to data such as network load, disk I / O and system load of the storage node (DN); the storage node (DN) is installed with node access services, node storage services and node cloud storage services, which provide interfaces for SDK clients to write data, and report system information to the metadata management server in real time.

[0047] Based on Figure 1 The video cloud storage system shown in the traditional video cloud storage system data transmission mode is described in detail in Figure 2 , and the video cloud storage system data transmission mode provided by the present application is described in Figure 3 .

[0048] Comparing Figure 2 and Figure 3 , it can be seen that the traditional video cloud storage system data transmission mode needs three network transmissions to realize data transmission between different services in the storage node, i.e., the first to third data transmission processes in Figure 2 . The video cloud storage system data transmission mode provided by the present application needs only one network transmission, because it uses a physical network card to realize video data access of the front-end device, i.e., the first data transmission process in Figure 3 , and uses a loopback network card to realize data transmission between different services in the storage node, i.e., the second to third data transmission processes in Figure 3 .

[0049] In summary, the video cloud storage system data transmission mode provided by the present application reduces the number of network transmissions and the consumption of network resources by adding a loopback network card.

[0050] Further, the present application improves and optimizes the use of the loopback network card in the video cloud storage system through an algorithm, saves network resources by using the loopback network card, has the characteristics of high transmission efficiency and does not occupy network resources, further optimizes the network transmission efficiency, and fully improves the performance of the video cloud storage system. For details, please refer to Figure 4 and Figure 5 , Figure 4 is a flowchart of an embodiment of the data transmission method provided by the present application, Figure 5 is a flowchart of an access data flow conversion strategy provided by the present application.

[0051] Figure 4 The data transmission method shown in the video cloud storage system is applied to the storage node in the video cloud storage system, and specifically relates to a data flow conversion strategy of the front-end device and the node access service in the storage node.

[0052] As Figure 4As shown, the data transmission method provided in the present application specifically includes the following steps:

[0053] Step S11: accessing the video data of the front-end device.

[0054] In the embodiment of the present application, after the front-end device network is connected, the device information is configured to the video cloud storage system. At this time, the access management service of the metadata management server takes over the front-end device. After the node access service of the storage node is started, it will periodically report the network load (NetLoad) to the access management service of the metadata management server. The access management service will allocate the front-end device such as a camera and a video recorder to a certain storage node according to the network load.

[0055] Specifically, the access management service sorts according to the network load (NetLoad) reported by the node, and preferentially allocates the front-end device to the node with low network load.

[0056] At the same time, the node access service maintains the data flow transfer mode field (Local: local, Net: network, Local_Net: both modes are performed at the same time), and reports the flow transfer mode and the network load to the access management service in real time according to the local network load. The node access service transfers the video data of the front-end device to the node storage service through the loopback network card according to the maintained flow transfer mode, that is, as shown in the following figure. Figure 3 The 2 data transmission processes.

[0057] Step S12: obtaining the system network load of the storage node.

[0058] In the embodiment of the present application, after the video data of the front-end device is accessed, the system network load changes, and the access node service senses the change and obtains the system network load of the node.

[0059] Step S13: based on the system network load, the video data is transmitted through the local loop and / or network transmission and stored to the disk of the storage node.

[0060] In the embodiment of the present application, when the system network load of the node is greater than the first preset threshold NetLoad H , the data is transferred to the node storage service through the local loop, the flow transfer mode field is modified to Local, and the real-time state is reported.

[0061] In other embodiments, if the network load of the node is NetLoad N , and is lower than the threshold NetLoad H , but is higher than the threshold NetLoad L , the node access service allocates the access point according to the proportion, and the insufficient access proportion is transferred in the form of local loop network.

[0062] Specifically, the node access service transmits the point flow of NetLoad N / (NetLoad H + NetLoad L ) in a network form, and the remaining point flow, i.e., 1-NetLoad N / (NetLoad H + NetLoad L ), is circulated in a local loop form.

[0063] In other embodiments, if the network load of the node is lower than a threshold value NetLoad L , all point data will be circulated to the node storage service in a network manner.

[0064] The system network load of the storage node is calculated as follows: NetLoad N = T d / T. Wherein, T is the theoretical bandwidth (MB / s), and T d is the data transmission amount (M) per unit time.

[0065] It should be noted that, in order to limit the flow in the network form, NetLoad H can also be set to a lower value, i.e., the local loop network address is preferred.

[0066] Further, as shown in Figure 5 , if the video cloud storage system has only one storage node, the video data of the front-end device can be directly transmitted to the node storage service in a local loop manner.

[0067] In the embodiments of the present application, the storage node accesses the video data of the front-end device; the system network load of the storage node is obtained; and the video data is stored to the disk of the storage node through local loop transmission and / or network transmission based on the system network load. Through the above manner, the cloud storage system utilizes the characteristics of high bandwidth and high transmission efficiency of the loop card, reduces the real network load, improves the network resource utilization, and improves the overall system performance.

[0068] Further, please continue to refer to Figure 6 and Figure 7 , Figure 6 is a flowchart of another embodiment of the data transmission method provided by the present application, Figure 7 is a flowchart of the cloud storage management service allocation strategy provided by the present application.

[0069] Figure 6The data transmission method shown is applied to a video cloud storage system, wherein the video cloud storage system comprises a metadata management server and a storage node, and particularly relates to a storage service and a cloud storage service data flow conversion strategy.

[0070] As shown in the figure, Figure 6 The data transmission method provided by the application specifically comprises the following steps:

[0071] Step S21: The metadata management server acquires load information of the storage node, and generates a data write mode according to the load information.

[0072] In the embodiment of the application, the metadata management server storage management service limits the data flow received by the node storage service according to the load balancing principle. Since the node storage service will accept data flow from the local loopback network and from the external network, it will timely report the data source and total amount (D) to the storage management service. At the same time, the node storage service will call the SDK client write data interface of the cloud storage to inform the SDK client of the data information. The SDK client will magnify the actual data flow to (N+M) / N times through EC (erasure coding) calculation. Wherein, N+M is the redundancy mode, that is, the file data block is divided into N data blocks and M check blocks.

[0073] At the same time, the SDK client requests the data write mode and the specific node position of the cloud storage management service, and the cloud storage management service returns the corresponding value to the SDK client. The return value includes the flow conversion mode (Local, Net or LocalandNet) and the flow conversion node position information. After obtaining the data flow conversion mode, the SDK client transmits the data to the node cloud storage service according to the flow conversion mode and the flow conversion node position information, and writes the data into the disk by the node cloud storage service. The cloud storage management service returns the flow conversion mode, that is, the data write mode to the SDK client in time through a certain strategy according to the comprehensive information reported by the node cloud storage service.

[0074] The node cloud storage service collects the disk number (disk_num), the average I / O load (io_load) and the system resource load (sys_load) of the node and reports them to the cloud storage management service. The cloud storage management service judges the data reported by the node cloud storage service, and distributes according to the mode of giving priority to local loopback flow conversion. The specific strategy is as shown in the figure. Figure 7

[0075] Specifically, if the video cloud storage system has only one node, the data is directly transmitted in the local loopback mode.

[0076] ​If the video cloud storage system has multiple nodes, the cloud storage management service judges the data reported by the node cloud storage service. If the average load of the system resources of the video cloud storage system as a whole (sys_load_avg) is higher than the threshold SYS_LOAD, the data is transmitted in the local loopback mode.

[0077] If the video cloud storage system has multiple nodes, and the average load of the system resources of the video cloud storage system as a whole (sys_load_avg) is lower than the threshold SYS_LOAD, as long as the storage node meets the condition that the number of disks is greater than N+M or the average io load of the disks is less than IO_LOAD, the data is transmitted to the node cloud storage service in the loopback network form, and then the node cloud storage service writes the data into the disk.

[0078] Step S22: The storage node obtains the data writing mode allocated by the metadata management server, and writes the video data of the front-end device into the disk according to the data writing mode.

[0079] The video cloud shortens the network path single machine EC optimization method described in the data transmission method of the present application. Compared with the prior art, the characteristics of high bandwidth and high transmission efficiency of the loopback network card are fully utilized, the real network load is reduced, the network resource utilization is improved, and the overall system performance is improved. In addition, the video data flow conversion strategy described in the above embodiment is based on the local loopback priority principle, and simultaneously performs rule judgment according to system related data such as network load, system load, and disk IO load, thereby further improving the network resource utilization of the system.

[0080] Specifically, the present application provides a video cloud shortens the network path single machine EC optimization method. In the environment of the video cloud storage, the traffic accessing the service needs to spend one unit of bandwidth to reach the storage service. The selection algorithm of the access management service selects the data transmission mode according to the system load and the network load.

[0081] If the loopback mode is used for transmission, the first level of network bandwidth resource can be saved. The network traffic from the storage service to the cloud storage service needs to pass through the EC amplification. Taking the 4+1 writing mode as an example, the network traffic is expanded to 1.25 times of the original, the selection algorithm of the storage service takes the loopback network as the priority, and the traffic transmission mode is obtained according to the system load, the disk load, and the number of disks, so that the second level of network bandwidth resource can be saved. The cloud storage service writes the data into the disk according to the number of nodes, the number of disks, and the load. Through the selection of the two-level algorithm, the network resource under the highest efficiency is reduced from the original 3.25 units of bandwidth to 1.25 units of bandwidth.

[0082] The application selects, through an algorithm, to circulate access service, storage service and cloud storage service data flow in the same machine, utilize the high bandwidth and high transmission efficiency characteristics of the loopback network card, and reduce network flow amplification and maximize the use of network resources for the entire video cloud cluster.

[0083] The application provides an optimization method of a video cloud shortened network path single machine EC, and front-end recording is uniformly managed by an access management service, different front-end device data streams are distributed to an access node service according to a load balancing principle, the access node service selects network transmission or local loopback data transmission to a storage service according to a transmission strategy distributed by the access management service, and the transmission strategy of the access node service is as follows.

[0084] a) judging the local network load, if the network load exceeds a threshold value NetLoadH, the loopback network is used to transmit data to the storage service.

[0085] b) judging the local network load, if the network load is lower than a threshold value NetLoad H , but higher than a threshold value NetLoad L , data is transmitted to the storage service according to a proportion of NetLoad N / (NetLoad H + NetLoad L ) through the network and a proportion of 1-NetLoad N / (NetLoad H + NetLoad L ) through the loopback.

[0086] c) judging the local network load, if the network load is lower than NetLoad L , data is transmitted to the storage service through the network.

[0087] The storage service data transmission strategy to the cloud storage service is as follows.

[0088] a) the storage service calls a cloud storage SDK write data interface, data is calculated by an EC algorithm of the SDK to expand the data to (N+M) / N times of the original data, and the loopback network is selected for the first time according to the system node quantity.

[0089] b) the cloud storage management service selects the loopback network for the second time according to the overall load of the system.

[0090] c) if the first and second selection conditions are not met, the cloud storage management service selects the loopback for the third time according to the disk quantity and the average load of the disk io, otherwise, the data needs to be transmitted in the form of the network.

[0091] Those skilled in the art can understand that the sequence of writing each step in the above method of the specific embodiment does not mean a strict execution sequence and does not constitute any limitation on the implementation process. The specific execution sequence of each step should be determined by its function and possible internal logic.

[0092] To implement the data storage method of the above-mentioned embodiments, the present application further provides a cloud storage system. For details, please refer to Figure 8 , Figure 8 is a structural schematic diagram of an embodiment of the cloud storage system provided by the present application.

[0093] The cloud storage system 500 of the embodiment of the present application comprises a memory 51 and a processor 52, wherein the memory 51 and the processor 52 are coupled.

[0094] The memory 51 is used to store program data, and the processor 52 is used to execute the program data to implement the data storage method described in the above-mentioned embodiments.

[0095] In the embodiment, the processor 52 can also be referred to as a CPU (Central Processing Unit). The processor 52 can be an integrated circuit chip with signal processing capability. The processor 52 can also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor 52 can also be any conventional processor or the like.

[0096] To implement the data storage method of the above-mentioned embodiments, the present application further provides a computer readable storage medium, such as Figure 9 As shown in the figure, the computer readable storage medium 600 is used to store program data 61, and the program data 61 is used to implement the data storage method as described in the above-mentioned embodiments when executed by the processor.

[0097] The present application further provides a computer program product, wherein the above-mentioned computer program product comprises a computer program, and the above-mentioned computer program is operable to make a computer execute the data storage method as described in the embodiments of the present application. The computer program product can be a software installation package.

[0098] The data storage method described in the above embodiments of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0099] The above description is only an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, based on the content of the present application specification and drawings, are also included in the patent protection scope of the present application.

Claims

1. A data storage method, characterized in that, The data storage method is applied to a storage node in a cloud storage system; the data storage method includes: Accessing video data from front-end devices; Obtain the system network load of the storage node; Based on the system network load, the video data is transmitted locally via loopback and / or transmitted over the network to the disk of the storage node; The step of storing the video data to the disk of the storage node via local loopback transmission and / or network transmission based on the system network load includes: When the system network load exceeds a first preset threshold, the video data is stored in the disk of the storage node via the local loopback transmission. When the system network load is less than a second preset threshold, the video data is transmitted to the disk of the storage node via the network; wherein the second preset threshold is less than the first preset threshold; When the system network load is greater than the second preset threshold and less than the first preset threshold, a portion of the video data is transmitted over the network and stored on the disk of the storage node according to the ratio of the system network load to the preset threshold, and the remaining video data is transmitted locally and stored on the disk of the storage node. The storage node includes a node access service and a node storage service. The node access service maintains a data flow mode field, which includes local loopback transmission mode, network transmission mode, or both simultaneously. The node access service updates the data flow method field in real time based on the system network load. Based on the data flow method field, the node access service transfers the video data to the node storage service through local loopback transmission and / or network transmission, so that the node storage service can write the data to the disk of the storage node.

2. The data storage method according to claim 1, characterized in that, The ratio of the system network load to the preset threshold is the ratio of the system network load to the sum of the first preset threshold and the second preset threshold.

3. The data storage method according to any one of claims 1 to 2, characterized in that, After the video data from the access front-end device is received, the data storage method further includes: Obtain the number of storage nodes in the cloud storage system; When the number of storage nodes is 1, the video data is stored on the disk of the storage node through the local loopback transmission.

4. A data storage method, characterized in that, The data storage method is applied to a cloud storage system, wherein the cloud storage system includes a metadata management server and storage nodes; the data storage method includes: The metadata management server obtains the load information of the storage node and generates a data writing method based on the load information; The storage node obtains the data writing method allocated by the metadata management server and writes the video data of the front-end device to the disk according to the data writing method; The load information includes the average system resource load of all storage nodes in the cloud storage system; The method for generating and writing data based on the load information includes: When the average load of the system resources exceeds a preset load threshold, the data writing method is set to local loopback transmission. The storage node obtains the data writing method allocated by the metadata management server, and writes the video data of the front-end device to the disk according to the data writing method, specifically including: The cloud storage system includes an SDK client; The storage node includes node storage service, node cloud storage service, and an interface for writing data to the SDK client; The node storage service receives the video data and calls the interface that provides data writing to the SDK client to transmit the video data to the SDK client; The SDK client sends a request to the metadata management server to obtain the streaming method of the video data; The metadata management server returns a value to the SDK client, wherein the return value includes the transfer method; After obtaining the streaming method, the SDK client transmits the video data to the node cloud storage service according to the streaming method, so that the node cloud storage service can write the video data to the disk.

5. The data storage method according to claim 4, characterized in that, The load information includes the number of disks in the storage node and the average disk load; The method for generating and writing data based on the load information includes: When the number of disks is greater than the storage threshold of the video data of the front-end device, or when the average disk load is less than the preset disk threshold, the data writing method is set to local loopback transmission. The storage threshold for the video data is determined by the segmented valid data blocks and the verification data blocks.

6. A cloud storage system, characterized in that, The cloud storage system includes a processor and a memory, wherein the memory stores program data, and the processor executes the program data to implement the data storage method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program data, which, when executed by a processor, is used to implement the data storage method according to any one of claims 1 to 5.

Images