Video storage method and related device

Abstract

The invention discloses a video storage method and a related device, and relates to the technical field of storage, a video storage management system comprises a global metadata management server and a plurality of distributed storage servers, a global storage service runs on the global metadata management server, a node storage service runs on each distributed storage server, and the node storage service runs on the plurality of distributed storage servers. The node storage service generates metadata corresponding to a sub-file after monitoring that the sub-file of the video file falls into a disk, the metadata comprises a summary and details, the summary comprises a corresponding relation between the video file and a distributed storage server, the summary and the details are stored in the disk for storing the sub-file, the summary is sent to the global storage service, and the video file is stored in the global storage service. The metadata of the video file is stored in a distributed and centralized mode, the distributed storage server corresponding to the video file is determined according to the hash value corresponding to the video identifier of the video file, and the single-point fault and the performance bottleneck caused by centralized metadata management and storage are solved.

Description

Technical Field

[0001] This application relates to the field of storage technologies, and in particular, to a video storage method and related devices. Background Art

[0002] In the field of audio and video, due to the improvement of content production efficiency and the continuous iteration of encoding technologies, there is an increasing amount of existing content, and the storage requirements for videos in different formats and qualities are increasing.

[0003] The current main storage system methods include decentralized storage and centralized storage. Among them, in the decentralized storage method, the stored content is located using hashing, completely abandoning metadata, and it is impossible to effectively manage space, resulting in space imbalance. In the centralized storage method, the metadata is centrally managed, and the specific data is distributedly stored. Its advantages are simple management and efficient access, and its disadvantages are the risk of single-point failure and possible performance bottlenecks in the central node. Summary of the Invention

[0004] In view of the above problems, this application provides a video storage method and related devices, which can not only solve the single-point failure and performance bottleneck caused by centralized metadata management storage, but also achieve load balancing using a global storage service. The specific solutions are as follows:

[0005] In a first aspect of this application, a video storage method is provided, which is applied to a node storage service running on each distributed storage server in a video storage management system. The video storage management system further includes a global metadata management server, and a global storage service runs on the global metadata management server. The video storage method includes:

[0006] After monitoring that a sub-file of a video file has been written to disk, generate metadata corresponding to the sub-file. The metadata includes a summary and details. The summary includes the correspondence between the video file and the distributed storage server. The distributed storage server corresponding to the video file is determined according to the hash value corresponding to the video identifier of the video file, and the disk storing the sub-file is determined according to the hash value corresponding to the file identifier of the sub-file;

[0007] Store the summary and details in the metadata on the disk storing the sub-file;

[0008] Send the summary in the metadata to the global storage service.

[0009] In a possible implementation, after monitoring that a sub-file of a video file has been written to disk, the video storage method further includes:

[0010] If the disk space for storing the sub-files is unbalanced, move the sub-files to other disks in the distributed storage server and record the location of the sub-file dump in the metadata.

[0011] In a possible implementation, the video storage method further includes:

[0012] Verify the correspondence between the video file and the distributed storage server according to the summary in the metadata;

[0013] Verify whether the video file is missing shards;

[0014] If shards are missing, initiate an alarm or patch.

[0015] In a possible implementation, the video storage method further includes:

[0016] In the case of detecting a disk failure, determine a target disk with disk remaining space greater than the threshold, and create a temporary directory on the target disk;

[0017] Use a soft link to point the mounted directory of the failed disk to the temporary directory;

[0018] Reconstruct the file structure of the temporary directory according to the storage directories in other disks in the distributed storage server except the failed disk.

[0019] In a possible implementation, after reconstructing the file structure of the temporary directory according to the storage directories in other disks in the distributed storage server except the failed disk, the video storage method further includes:

[0020] Monitor the pressure parameter of the target disk;

[0021] If the pressure parameter of the target disk is within the first preset range, initiate patching and update the metadata corresponding to the temporary directory;

[0022] If the pressure parameter of the target disk is within the second preset range, keep it unchanged;

[0023] If the pressure parameter of the target disk is within the third preset range, migrate the newly created files under the temporary directory to other disks in the distributed storage server and update the metadata corresponding to the temporary directory;

[0024] Wherein, the pressure parameter in the first preset range is less than the pressure parameter in the second preset range, and the pressure parameter in the second preset range is less than the pressure parameter in the third preset range.

[0025] In a possible implementation, after reconstructing the file structure of the temporary directory according to the storage directories in other disks of the distributed storage server except the faulty disk, the video storage method further includes:

[0026] When it is monitored that the faulty disk is changed from a soft link to a physical disk or a recovery instruction for the faulty disk is received, move all the contents in the temporary directory to the disk after fault recovery;

[0027] If the newly created files under the temporary directory are migrated to other disks in the distributed storage server, migrate the newly created files under the temporary directory back to the disk after fault recovery.

[0028] In a possible implementation, the video storage method further includes:

[0029] When the HTTP service fails to locate the target video file in the file system according to the preset hash location rule, determine the storage directory of the target video file based on the local metadata;

[0030] If the storage directory of the target video file cannot be determined based on the local metadata, send a query request to the global storage service to enable the global storage service to determine the storage directory of the target video file.

[0031] In a possible implementation, the video storage method further includes:

[0032] Before the new distributed storage server goes online, send a query request to the global storage service;

[0033] If the new distributed storage server is a replacement server for the original distributed storage server, receive the metadata of the original distributed storage server fed back by the global storage service;

[0034] If the new distributed storage server is an expansion server, receive the metadata to be expanded fed back by the global storage service;

[0035] Initiate a registration request to the global storage service;

[0036] Receive the registration response information fed back by the global storage service, and send a summary of the stored metadata to the global storage service.

[0037] A second aspect of the present application provides a distributed storage server, including at least one processor and a memory connected to the processor, where:

[0038] The memory is used to store computer programs;

[0039] The processor is used to execute the computer program, so that the distributed storage server can implement the video storage method in the first aspect or any implementation manner of the first aspect.

[0040] The third aspect of this application provides a video storage management system, including: a global metadata management server and multiple distributed storage servers as described in the second aspect above;

[0041] A global storage service runs on the global metadata management server;

[0042] A node storage service runs on each of the distributed storage servers.

[0043] The fourth aspect of this application provides a computer program product, including computer-readable instructions, which, when running on a distributed storage server, enable the distributed storage server to implement the video storage method in the first aspect or any implementation manner of the first aspect.

[0044] By means of the above technical solutions, a video storage method and related devices provided by this application. The video storage management system includes a global metadata management server and multiple distributed storage servers. A global storage service runs on the global metadata management server, and a node storage service runs on each distributed storage server. After the node storage service monitors that the sub-files of the video file are written to disk, it generates metadata corresponding to the sub-files. The metadata includes a summary and details. The summary includes the correspondence between the video file and the distributed storage server. By storing the summary and details on the disk storing the sub-files and sending the summary to the global storage service, it realizes the distributed plus centralized storage of the metadata of the video file. The distributed storage server corresponding to the video file is determined according to the hash value corresponding to the video identifier of the video file, which can solve the single point of failure and performance bottleneck caused by centralized metadata management and storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Combined with the drawings and referring to the following specific embodiments, the above and other features, advantages and aspects of the embodiments of the present disclosure will become more obvious. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the original components and elements are not necessarily drawn to scale.

[0046] Figure 1 It is a schematic diagram of a system architecture provided by an embodiment of this application;

[0047] Figure 2 It is a schematic diagram of the structure of a video storage management system provided by an embodiment of this application;

[0048] Figure 3 It is a schematic diagram of the process of a video storage method provided by an embodiment of this application;

[0049] Figure 4 Schematic diagram of content addressing provided by the embodiments of the present application;

[0050] Figure 5 Schematic diagram of a sub-file storage process provided by the embodiments of the present application;

[0051] Figure 6 Schematic diagram of a disk drop failure handling process provided by the embodiments of the present application;

[0052] Figure 7 Schematic diagram of a load balancing processing flow after disk drop provided by the embodiments of the present application;

[0053] Figure 8 Schematic diagram of a disk failure recovery process provided by the embodiments of the present application;

[0054] Figure 9 Schematic diagram of a video file access process provided by the embodiments of the present application;

[0055] Figure 10 Schematic diagram of online expansion traffic provided by the embodiments of the present application. Detailed implementation manners

[0056] The embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. The terms used in the implementation part of the present application are only used to explain the specific embodiments of the present application, rather than intended to limit the present application.

[0057] The embodiments of the present application will be described below with reference to the accompanying drawings. Those skilled in the art will know that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0058] The terms "first", "second", etc. in the specification and claims of the present application and the above accompanying drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances, which is only a way of distinguishing when describing objects with the same attributes in the embodiments of the present application. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product or device comprising a series of units does not have to be limited to those units, but may include other units not clearly listed or inherent to these processes, methods, products or devices.

[0059] Please refer to Figure 1The schematic diagram of the system architecture shown. In the monitoring and management layer of the video storage management system, there is a global metadata management server. A global storage service runs on the global metadata management server, and a log management service and a graphical monitoring service can also run. The global metadata management server can include one or more servers, that is, the global storage service can also be a cluster service.

[0060] The storage layer in the video storage management system includes multiple distributed storage servers. A node storage service runs on each distributed storage server. The distributed storage servers can be distributed in different computer rooms, such as Figure 1 shown, multiple distributed storage servers are distributed in Computer Room A and Computer Room B. Each distributed storage server corresponds to a unique node identifier (NodeID). Each distributed storage server includes multiple disks, and each disk corresponds to a storage directory respectively. Each video file corresponds to a video identifier (FileID). A video file includes multiple sub-files, and each sub-file corresponds to a file identifier respectively.

[0061] Video files are uploaded by the client. After passing through the CDN (Content Delivery Network), the access layer performs a hash operation on the video identifier of the video file to determine a uniquely matching node identifier (NodeID). Specifically, for the storage of the sub-files of the video file, a hash operation is performed on the file identifier of the sub-file to determine the unique storage directory, and the sub-file is saved in "NodeID / storage directory / ".

[0062] An embodiment of this application provides a video storage management system. Please refer to Figure 2 , the video storage management system includes: a global metadata management server 201 and multiple distributed storage servers 202 ( Figure 2 illustrated with 3 distributed storage servers as an example). Among them, a global storage service runs on the global metadata management server 201, and a node storage service runs on each distributed storage server 202.

[0063] An embodiment of this application provides a video storage method, which is applied to the node storage service running on each distributed storage server in the video storage management system. The video storage method of the embodiment of this application will be introduced in detail below with reference to the accompanying drawings.

[0064] Refer to Figure 3 , Figure 3 is a schematic flowchart of a video storage method provided by an embodiment of this application. As Figure 3 shown, a video storage method provided by an embodiment of this application may include steps 301-303. These steps will be described in detail below.

[0065] 301: After monitoring the sub - files of a video file being written to disk, generate metadata corresponding to the sub - files. The metadata includes a summary and details.

[0066] The distributed storage server corresponding to the video file is determined according to the hash value corresponding to the video identifier of the video file, and the disk storing the sub - files is determined according to the hash value corresponding to the file identifier of the sub - files. Exemplarily, as Figure 4 shown in the content - addressed schematic diagram, the node storing the video file is located as Node A according to the hash value of the video ID, and then the disks storing the sub - files are located as Disk 1, Disk 2, and Disk 12 according to the hash value of the sub - file ID.

[0067] The summary is used to describe the basic attributes of the video file. For example, the summary includes the correspondence between the video file and the distributed storage server.

[0068] In a possible implementation, the details include a sub - file part and a redundant backup part. The sub - file part is used to describe sub - file information, such as which files are in this directory and their respective states. The redundant backup part is used to save the sub - file information of other storage directories.

[0069] The summary and details are combined into a metadata file and saved at the lowest level of the video directory, in the same directory as the sub - files.

[0070] Exemplarily, a metadata file of meta.json has the following file format:

[0071] The metadata file of meta.json has the following file format:

[0072] {

[0073] "Summay": { / / Summary data

[0074] / / Data for patching

[0075] "FileID": "FSFDDSFS", "Main": " / c1 / xxx / xvd / xxx.m3u8", "NodeID": 1231, "Total": 12, "Size": 123333},

[0076] / / Description of sub - files in this directory. This data only saves the files that need to be saved in this directory according to the hash. The value of 1 means it is in the local, otherwise it is saved in another location and the specific location is pointed out.

[0077] "Local": {"12.ts": 1, "24.ts": " / data1", "36.ts": "nodeid / data1 / xxxx / 36.ts"},

[0078] / / Redundant backup

[0079] “Recovery” : {“ / data11” : {“11.ts” : 1}, “ / data1” : {“1.ts” : 1}}

[0080] }’

[0081] Among them, “Summay” represents the summary part, “FileID” represents the video identifier; “Main” represents the main file description, that is, the sub - file description; "Total" represents the number of sub - files of the video file; "Size" represents the storage size.

[0082] “Local” represents the sub - file part.

[0083] “Recovery” represents the redundant backup part.

[0084] 302: Store the summary and details in the metadata to the disk of the storage sub - file;

[0085] The node storage service maintains the consistency between the metadata and the video data. When changes occur in the relevant storage directories on the distributed storage server, the corresponding metadata will be updated in a timely manner. For example, when it is monitored that a new file / data1 / AAA / BBB / CCC.m3u8 is added to the storage directory / data1, the node storage service will create the corresponding metadata meta.json under / data1 / AAA / BBB / for this video, and determine whether to back up the metadata to the same path in other storage directories according to the pre - set redundancy policy.

[0086] It should be noted that different storage directories on the same distributed storage server have the same directory structure. For example, if the disk / data1 has the AAA / BBB directory, then other disks / data2, / data3, etc. must have the same AAA / BBB directory. Whether there is real data in the directory or not, the relevant disk metadata needs to be saved so that after a certain storage directory fails, it can be recovered according to the redundant metadata in other directories.

[0087] 303: Send the summary in the metadata to the global storage service.

[0088] By synchronizing a copy of the summary part in the metadata to the global metadata management service, the global metadata management service can establish the corresponding relationship between the video file and the distributed storage server according to this summary information, that is, the correspondence table between the video ID and the NodeID. When the file access service cannot locate the file through the hash value, the global storage service can locate the real location of the video file according to the summary in the metadata.

[0089] The video ID and NodeID here can have a one-to-many or many-to-many relationship, that is, a video can be stored on only one distributed storage server (i.e., the NodeID is unique), or it can be stored on multiple distributed storage servers, which needs to be determined according to different backup levels. If the video file is stored on only one distributed storage server, then the NodeID and the video ID must be determined by hash algorithm positioning.

[0090] A video storage method disclosed in this embodiment realizes distributed and centralized storage of the metadata of video files, determines the distributed storage server corresponding to the video file according to the hash value corresponding to the video identifier of the video file, and solves the single point of failure and performance bottleneck caused by centralized metadata management and storage.

[0091] In a possible implementation, the node storage service also has a load balancing function. As Figure 5 shown in the schematic diagram of the sub-file storage process, the node distribution service also runs on the distributed storage server. After the node distribution service downloads the sub-file and writes the sub-file to the disk, the node storage service listens for changes in the file system on the distributed storage server. When the sub-file is written to the disk, the node storage service generates metadata based on the sub-file and the video file and saves it in the video directory, and sends the summary (meta summary) in the metadata, that is, the correspondence between the video file and the distributed storage server, to the global storage service. When generating the metadata (meta), the load situation of the disk will be evaluated. If it is found that the disk space is unbalanced, the sub-file can be moved to the storage directory of other disks in the distributed storage server, and the position of the sub-file dump will be recorded in the metadata. After the dump is completed, the summary information of the video will be reported to the global storage service to ensure the consistency of the metadata and the video file while realizing load balancing.

[0092] In a possible implementation, in order to further ensure the consistency of the metadata and the video file, the node storage service periodically verifies the metadata stored in the distributed storage server. For example, the metadata stored in the distributed storage server is verified every day. Specifically, the correspondence between the video file and the distributed storage server is verified according to the summary in the metadata. If the node identifier of the distributed storage server corresponding to the hash value calculated according to the video identifier in the summary is inconsistent with the local identifier, an alarm will be initiated.

[0093] When storing video files such as HLS (HTTP Live Streaming) and MPEG-DASH (Dynamic Adaptive Streaming over HTTP), it is also possible to verify whether there are missing segments. For example, verify whether there are missing sub-files corresponding to the sub-file parts in the distributed storage server according to the sub-file parts in the metadata. If there are missing segments, an alarm or patch will be initiated. During the patching process, determine the video file corresponding to the missing sub-file according to the summary part in the metadata, and then query the global storage service to check whether redundant data is saved. If redundant data is saved, patch directly according to the redundant data. If redundant data is not saved, patch according to the external protocol, usually pulling from the CDN.

[0094] Disks in the distributed storage server may fail. After the disk corresponding to the storage directory fails, the node storage service will recover data from the metadata redundancy of other storage directories and backfill the relevant files to ensure that the video files can be used normally.

[0095] In a possible implementation, when the node storage service running on the distributed storage server detects a disk failure, it determines a target disk with a remaining disk space greater than the threshold, creates a temporary directory on the target disk, then uses a soft link to point the mount directory of the failed disk to the temporary directory, and then reconstructs the file structure of the temporary directory according to the storage directories on other disks except the failed disk in the distributed storage server.

[0096] Exemplarily, please refer to Figure 6 the schematic diagram of the disk failure handling process shown, and the disk failure handling process specifically includes the following steps 601-607:

[0097] 601: Detect a disk failure, such as the / data12 disk fails;

[0098] 602: Build a temporary directory (such as / data12t) on other disks, and generate a soft link of / data12 pointing to the temporary directory / data12t;

[0099] To ensure load balancing, a disk with more remaining disk space will be selected as the target disk, and a temporary directory will be created on the target disk. For example, the disk with a remaining disk space greater than the threshold is determined as the target disk, and the threshold is set according to the actual application scenario.

[0100] 603: Traverse the video directories of adjacent disks ( / data1 or / data2), and obtain the content of / data12 from meta.json;

[0101] ForFigure 4 Taking the distributed storage server shown as an example, if / data12 has a disk failure, the distributed storage server only has / data1 and / data2 left. By traversing / data1 or / data2, the metadata file meta.json stored in the same directory as / data12 can be obtained, and the content of / data12 can be determined from the sub-file part in meta.json.

[0102] 604: Restore the video content on / data12 and write it into the temporary directory;

[0103] 605: The file type on / data12 in meta.json;

[0104] If the file type on / data12 in meta.json is a file saved on / data12, that is, the file on / data12 in meta.json is saved in the faulty disk / data12 and cannot be written directly, then execute 606: Modify the corresponding file in meta.json to be a file to be patched;

[0105] If the file type on / data12 in meta.json is a file saved in other locations, then execute 607: Write directly.

[0106] If the file type on / data12 in meta.json is a file saved in other locations, directly write the file in other locations into the temporary directory.

[0107] In a possible implementation, in order to ensure load balancing after a disk failure, after reconstructing the file structure of the temporary directory, the pressure parameter of the target disk can also be monitored. If the pressure parameter of the target disk is within the first preset range, initiate patching and update the metadata corresponding to the temporary directory; if the pressure parameter of the target disk is within the second preset range, keep it unchanged; if the pressure parameter of the target disk is within the third preset range, migrate the newly created files in the temporary directory to other disks in the distributed storage server and update the metadata corresponding to the temporary directory; where the pressure parameter in the first preset range is less than the pressure parameter in the second preset range, and the pressure parameter in the second preset range is less than the pressure parameter in the third preset range, that is, the pressure parameter of the target disk within the first preset range indicates that the target disk has a small pressure, the pressure parameter of the target disk within the second preset range indicates that the target disk has a moderate pressure, and the pressure parameter of the target disk within the third preset range indicates that the target disk has a large pressure.

[0108] Exemplarily, please refer to Figure 7 the schematic diagram of the load balancing processing flow after a disk failure shown, and the load balancing processing flow after a disk failure specifically includes the following steps 701 - 705:

[0109] 701: Monitor the temporary disk where the newly created temporary directory / data12t is located after disk loss;

[0110] That is, monitor the target disk where the temporary directory is located.

[0111] 702: Comprehensively measure the pressure of this temporary disk;

[0112] For example, measure the pressure parameters of the target disk. The pressure parameters can be the remaining disk space, disk space utilization rate, etc.

[0113] If the pressure is small, execute 703: Patch and update meta.json;

[0114] Corresponding to the above step 606, since the pressure parameters of the target disk will change with file access and storage, when the pressure of the target disk is small, the sub-files to be patched in the above step 606 can be patched. That is, determine the video file corresponding to the missing sub-file according to the summary part in the metadata, and then query the global storage service to check if redundant data is saved. If redundant data is saved, directly patch according to the redundant data. If redundant data is not saved, patch according to the external protocol, usually pulling from the CDN.

[0115] If the pressure is moderate, execute 704: Keep unchanged;

[0116] If the pressure is large, execute 705: Migrate the newly created files in the / data12t directory to other disks and update meta.

[0117] Similarly, since the pressure parameters of the target disk will change with file access and storage, when the pressure of the target disk is large, ensure load balancing by migrating the newly created files in the / data12t directory to other disks and updating meta.

[0118] After the failed disk is restored and remounted after maintenance, it is necessary to restore the data of the failed disk. In one possible implementation, when it is monitored that the failed disk changes from a soft link to a physical disk or a restoration instruction for the failed disk is received, that is, when the failed disk is restored and remounted, move all the contents in the temporary directory to the disk after failure restoration. If the newly created files in the temporary directory are migrated to other disks in the distributed storage server, migrate the newly created files in the temporary directory back to the disk after failure restoration. If there are patches to be made in the temporary directory, perform patching to ensure the consistency of the video files and metadata in the restored disk.

[0119] Exemplarily, please refer to Figure 8 the schematic diagram of the disk failure recovery process shown. The disk failure recovery process specifically includes the following steps 801 - 804:

[0120] 801: Monitor the file system and detect that / data12 has been changed from a soft link to a physical disk, or receive a restoration instruction for / data12;

[0121] Similar to the above example, / data12 is a faulty disk.

[0122] 802: Move all the contents in the original temporary directory / data12t to the new / data12;

[0123] 803: Migrate back the files that were temporarily stored in / data12 and elsewhere in the original video meta.json, and update the corresponding meta.json;

[0124] Corresponding to step 705 above, migrate back the files that were migrated to other disks due to excessive pressure on the target disk to / data12, and update meta.json to ensure the consistency between the video files in / data12 and meta.json.

[0125] 804: Patch the data that needs patching in meta.json.

[0126] Corresponding to step 606 above, if there is data to be patched in meta.json, patch it. The patching process is the same as the patching process in the above embodiments and will not be elaborated here.

[0127] The embodiments of the present application also provide a video access method. Please refer to Figure 9 the schematic diagram of the video file access process shown. After the HTTP service receives a request for consistent hashing to a node, according to the preset hash positioning rule, that is, locate in the file system according to the hash value of the target video file. If the target video file is not found, query the node storage service running on the distributed storage server corresponding to the hash value of the target video file. The node storage service determines the storage directory of the target video file based on the local metadata, that is, query the video identifier of the target video file in the local metadata. If the video identifier of the target video file is not queried, the storage directory of the target video file cannot be determined based on the local metadata, and a query request is sent to the global storage service, so that the global storage service gives a conclusion according to the corresponding relationship between the video identifier in the metadata and the distributed storage server, either that there is no target video file or the specific location of the target video file.

[0128] The video storage management system also supports online expansion. By adding new distributed storage servers (hereinafter referred to as new nodes) to the video storage management system, the new distributed storage servers can be replacement servers (hereinafter referred to as replacement nodes) for the original distributed storage servers (hereinafter referred to as original nodes), and the new distributed storage servers can also be expansion servers (hereinafter referred to as expansion nodes). In a possible implementation, the online expansion process includes the following steps 901-905:

[0129] 901: Before the new node goes online, send a query request to the global storage service;

[0130] The new node is the distributed storage server added to the video storage management system.

[0131] 902: If the new node is a replacement node for the original node, receive the metadata of the original node fed back by the global storage service;

[0132] The original node is the original distributed storage server of the video storage management system.

[0133] 903: If the new node is an expansion node, receive the metadata to be expanded fed back by the global storage service;

[0134] That is, the original node will not go offline, and the new node is a newly added node.

[0135] 904: Send a registration request to the global storage service;

[0136] 905: Receive the registration response information fed back by the global storage service, and send a summary of the stored metadata to the global storage service.

[0137] Exemplarily, such as Figure 10Schematic diagram of the online expansion process shown. Before the new node goes online, the new node storage service queries its own video content from the global storage service. If the new node is a replacement for an existing node, it only needs to receive the metadata of the existing node fed back by the global storage service, construct the storage according to the metadata, and the storage directory in the new node is the same as that of the existing node. If the new node is an expansion node, it is necessary to submit the entire batch of expansion nodes to the global storage service. After the global storage service recalculates the hash based on the node identifiers of the existing distributed storage servers, it feeds back the metadata of the video files belonging to the expansion nodes to the expansion nodes. These video files are currently stored in other distributed storage servers, that is, the metadata to be expanded is fed back to the expansion nodes. The expansion nodes construct the storage according to the metadata to be expanded, establish their own storage directories and metadata meta. It should be noted that the initial saved meta information is all dump addresses, that is, redirected to the nodes where the original video data is saved. After the new node establishes its own metadata, it sends a registration request to the global storage service. After verification, the global storage service incorporates the new node into management, and then the new node reports its existing metadata to the global storage service, and the global storage service updates the data address based on the report.

[0138] In summary, the video storage method provided by the embodiment of the present application maintains the consistency between metadata and video data, supports fault recovery, file addressing, global retrieval, load balancing, and online expansion, avoids the risk of single-point failure, and improves system performance.

[0139] The embodiment of the present application also provides a distributed storage server, including at least one processor and a memory connected to the processor, wherein:

[0140] The memory is used to store computer programs;

[0141] The processor is used to execute the computer program so that the distributed storage server can implement any video storage method provided by the embodiment of the present application.

[0142] The embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions run on a distributed storage server, the distributed storage server is enabled to implement any video storage method provided by the embodiment of the present application.

[0143] The embodiment of the present application also provides a computer-readable storage medium. The storage medium carries one or more computer programs. When the one or more computer programs are executed by a distributed storage server, the distributed storage server can be enabled to implement any video storage method provided by the embodiment of the present application.

[0144] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationships between the modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines.

[0145] Through the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or dedicated circuits. However, for this application, in more cases, software program implementation is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disc of a computer, and includes several instructions to enable a computer device (which can be a personal computer, training device, or network device, etc.) to execute the methods described in various embodiments of this application.

[0146] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0147] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center by wired means (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless means (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device or a data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

Claims

1. A video storage method, characterized in that: The video storage management system is applied to a node storage service running on each distributed storage server in the video storage management system, wherein the video storage management system further comprises a global metadata management server, and the global metadata management server runs a global storage service. The video storage method comprises: After monitoring that a sub-file of a video file is stored on a disk, metadata corresponding to the sub-file is generated, the metadata including a summary and details, the summary including a correspondence between the video file and a distributed storage server, the distributed storage server corresponding to the video file is determined according to a hash value corresponding to a video identifier of the video file, and the disk storing the sub-file is determined according to a hash value corresponding to a file identifier of the sub-file; storing the summary and details in the metadata to a disk storing the sub-file; The summary in the metadata is sent to the global storage service.

2. The video storage method according to claim 1, characterized in that: After monitoring that the sub-file of the video file is stored on the disk, the video storage method further includes: If the disk space storing the sub-file is uneven, the sub-file is moved to other disks in the distributed storage server, and the location where the sub-file is dumped is recorded in the metadata.

3. The video storage method according to claim 1, characterized in that: The video storage method further includes: Verify the correspondence between the video file and the distributed storage server according to the summary in the metadata; Check whether the video file is missing segments; If a fragment is missing, an alarm is issued or a patch is added.

4. The video storage method according to claim 1, characterized in that: The video storage method further includes: In the case of a disk failure being detected, determining a target disk where the remaining disk space is greater than a threshold, and creating a temporary directory on the target disk; Use a soft link to point the mount directory of the failed disk to the temporary directory; The file structure of the temporary directory is rebuilt according to the storage directories in other disks in the distributed storage server except the failed disk.

5. The video storage method according to claim 4, characterized in that: After rebuilding the file structure of the temporary directory according to the storage directories in other disks except the failed disk in the distributed storage server, the video storage method further includes: monitoring a pressure parameter of the target disk; If the pressure parameter of the target disk is within a first preset range, initiating a patch and updating metadata corresponding to the temporary directory; If the pressure parameter of the target disk is within the second preset range, it remains unchanged; If the pressure parameter of the target disk is within a third preset range, migrating the newly created files in the temporary directory to other disks in the distributed storage server and updating metadata corresponding to the temporary directory; The pressure parameter in the first preset range is smaller than the pressure parameter in the second preset range, and the pressure parameter in the second preset range is smaller than the pressure parameter in the third preset range.

6. The video storage method according to claim 4, characterized in that: After rebuilding the file structure of the temporary directory according to the storage directories in other disks except the failed disk in the distributed storage server, the video storage method further includes: When it is detected that the failed disk is changed from a soft link to a physical disk or a recovery instruction of the failed disk is received, all contents in the temporary directory are moved to the recovered disk; If the newly created files in the temporary directory are migrated to other disks in the distributed storage server, the newly created files in the temporary directory are migrated back to the disk after the failure is restored.

7. The video storage method according to claim 1, characterized in that: The video storage method further includes: When the HTTP service fails to locate the target video file in the file system according to the preset hash positioning rule, determining the storage directory of the target video file according to the local metadata; If the storage directory of the target video file cannot be determined based on the local metadata, a query request is sent to the global storage service to enable the global storage service to determine the storage directory of the target video file.

8. The video storage method according to claim 1, characterized in that: The video storage method further includes: Before the new distributed storage server goes online, a query request is sent to the global storage service; If the new distributed storage server is a replacement server of the original distributed storage server, receiving metadata of the original distributed storage server fed back by the global storage service; If the new distributed storage server is an expansion server, receiving metadata to be expanded fed back by the global storage service; Initiate a registration request to the global storage service; Receive registration response information fed back by the global storage service, and send a summary of the stored metadata to the global storage service.

9. A distributed storage server, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the distributed storage server can implement the video storage method according to any one of claims 1 to 8.

10. A video storage management system, characterized in that: include: A global metadata management server and a plurality of distributed storage servers as claimed in claim 9; The global metadata management server runs a global storage service; A node storage service runs on each of the distributed storage servers.

11. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on a distributed storage server, the distributed storage server implements the video storage method according to any one of claims 1 to 8.

Images