Video data storage methods, devices, computer equipment and storage media

Abstract

This application belongs to the field of intelligent storage and relates to a video data storage method, including: acquiring a file to be uploaded; generating corresponding file data based on the file to be uploaded; determining whether the file to be uploaded is an appendable object based on the file data; if the file to be uploaded is determined to be an appendable object, acquiring the storage protocol of the file data and the append mode of the file to be uploaded; segmenting the file to be uploaded according to the storage protocol of the file data, and generating at least one data block with a storage space not exceeding a threshold and a corresponding data block identifier; creating a file data storage pool and a data block storage pool; storing the file data in the file data storage pool through a Ceph distributed storage system; and allocating corresponding data block storage pools for data blocks with storage spaces not exceeding the threshold based on the data identifiers. This application also provides a video data storage device, computer equipment, and storage medium. This reduces the resources occupied by video storage.

Description

Technical Field

[0001] This application relates to the field of intelligent storage, and in particular to a video data storage method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the increasing prevalence of big data analytics scenarios, the need for append-only writes in object storage for archived data is becoming more apparent. For example, the booming live streaming industry and log analysis, among other big data scenarios, require the continuous storage of archived / analyzed data on storage servers. Standard Simple Storage Service (S3) stores data using an overwrite approach. This means that when a user stores a file with the same name, the backend storage system deletes the previous data and creates a new file. Consequently, the application must manage a large list of objects. Due to the irregular nature of business operations, file sizes may vary significantly. In extreme cases, a massive list of small files may be maintained. When retrieving a portion of data, this necessitates sequentially retrieving numerous small files, resulting in a large number of small I / O operations, a sharp increase in IOPS, and a subsequent performance degradation. Summary of the Invention

[0003] The purpose of this application is to provide a video data storage method, apparatus, computer equipment, and storage medium that reduces the resources occupied by video storage.

[0004] To address the aforementioned technical problems, this application provides a video data storage method, employing the following technical solution:

[0005] Get the file to be uploaded;

[0006] Generate corresponding file data based on the file to be uploaded;

[0007] Determine whether the file to be uploaded is an appendable object based on the file data;

[0008] If it is determined that the file to be uploaded is an appendable object, then the storage protocol of the file data and the append mode of the file to be uploaded are obtained;

[0009] The file to be uploaded is segmented according to the file data storage protocol, and at least one data block with a storage space not exceeding a threshold and a corresponding data block identifier is generated.

[0010] Create a file data storage pool and a data block storage pool;

[0011] The file data is stored in the file data storage pool using the Ceph distributed storage system;

[0012] The corresponding data block storage pool is allocated according to the data identifier of the data block whose storage space does not exceed the threshold.

[0013] Furthermore, the step of determining whether the file to be uploaded is an appendable object based on the file data specifically includes:

[0014] Read the file data;

[0015] The file identifier in the file data is matched with a preset identifier, the file identifier being used to indicate the file type, and the preset identifier including an appendable identifier and a non-appendable identifier;

[0016] When the file identifier matches the preset appendable identifier, the file to be uploaded is determined to be an appendable object;

[0017] When the file identifier matches the preset non-appendable identifier, the file to be uploaded is determined to be a non-appendable object.

[0018] Furthermore, the file data includes at least a file data identifier, and the step of segmenting the file to be uploaded according to the file data storage protocol and generating at least one data block with a storage space not exceeding a threshold and a corresponding data identifier specifically includes:

[0019] If the storage protocol of the file data is a multi-copy protocol, then the file to be uploaded will be divided into at least one data block with a storage space not exceeding the threshold.

[0020] A data block identifier is generated for each data block based on the file data identifier number and the fragment number corresponding to the data block;

[0021] Alternatively, if the storage protocol for the file data is erasure coding (EC) protocol, then the end of the file to be uploaded is padded to align with EC to obtain the target file to be uploaded;

[0022] The target file to be uploaded is split into at least one data block whose storage space does not exceed a threshold.

[0023] A data block identifier is generated for each data block based on the file data identifier number and the fragment number corresponding to the data block.

[0024] Furthermore, the step of allocating the corresponding data block storage pool based on the data identifier indicating that the storage space does not exceed the threshold specifically includes:

[0025] Obtain the identifier of the data block storage pool;

[0026] The object identifier object_ID is obtained by hashing the data block identifier using the Ceph distributed storage system and taking the modulo operation.

[0027] The data block storage pool identifier Pool_ID is obtained by hashing the identifier number of the data block storage pool through the Ceph distributed storage system.

[0028] Merge the Pool_ID and the object_ID to obtain the PG_ID;

[0029] Storage pools are allocated for each data block and file data that does not exceed the threshold, using the PG_ID.

[0030] Furthermore, after the step of obtaining the file to be uploaded, the method further includes:

[0031] Define CGroup hierarchical rules in the storage pool;

[0032] Determine whether the Ceph distributed storage system process is started. If the Ceph distributed storage system process is started, load the CGroup hierarchical rules and start the CGroup hierarchical rules of the Ceph distributed storage system process.

[0033] Determine whether the Ceph distributed storage system process is shut down. If the Ceph distributed storage system process is shut down, then clear the CGroup hierarchical rules.

[0034] By mounting the storage pool in the CGgroup, system resources are controlled through the CGroup to allocate resources to the Ceph distributed storage system.

[0035] Furthermore, after the steps of creating the file data storage pool and the data block storage pool, the method further includes:

[0036] When a restart command is received, the receiving of the file to be uploaded is stopped and uninstallation information is generated.

[0037] The radosgw mount point of the Ceph partitioned storage system is unloaded using the unloading information.

[0038] Once all the fuse mount points are unloaded, stop the file system service of the Ceph distributed storage system and generate redirection information;

[0039] The restart command is redirected to the restart service to restart the Ceph distributed storage system.

[0040] Furthermore, prior to the step of obtaining the file to be uploaded, the following steps are also included:

[0041] Obtain the default deployment parameters of Ceph distributed storage system components. The default parameters include at least object storage gateway parameters, interface class parameters, authentication class parameters, data partitioning parameters, object storage unit parameters, and monitoring service parameters.

[0042] Obtain the deployment instructions for the Ceph distributed storage system;

[0043] Obtain the deployment instructions and deployment parameters, and generate corresponding scheduling instructions for the Ceph distributed storage system resources by combining the deployment instructions and deployment parameters in a fixed format;

[0044] The scheduling instructions are used to retrieve the software package resources and function library resources of the Ceph distributed storage system.

[0045] By selecting the binary mode transmission of the Ceph distributed storage system, the Ceph distributed storage system retrieves the software package resources and function library resources of the Ceph distributed storage system and transmits them to the target K-UX operating system;

[0046] Set the environment configuration parameters of the Ceph distributed storage system, and read the environment configuration parameters through the Ceph distributed storage system;

[0047] Deploy the environment configuration parameters to the Ceph distributed storage system;

[0048] Real-time monitoring and determination of whether the Ceph distributed storage system resources and their installation status are faulty;

[0049] If the Ceph distributed storage system resource fails, the installation of the Ceph distributed storage system resource will be reset to redeploy the Ceph distributed storage system.

[0050] To address the aforementioned technical problems, this application also provides a video data storage device, which employs the following technical solution:

[0051] The file acquisition module is used to acquire the file to be uploaded;

[0052] The file data generation module is used to generate corresponding file data based on the file to be uploaded.

[0053] An append determination module is used to determine whether the file to be uploaded is an appendable object based on the file data;

[0054] The protocol acquisition module is used to acquire the storage protocol of the file data and the append mode of the file to be uploaded if it is determined that the file to be uploaded is an appendable object.

[0055] The file segmentation module is used to segment the file to be uploaded according to the file data storage protocol, and generate at least one data block with a storage space not exceeding a threshold and a corresponding data block identifier.

[0056] The storage pool creation module is used to create file data storage pools and data block storage pools;

[0057] The file data storage module is used to store the file data in the file data storage pool through the Ceph distributed storage system;

[0058] The data block storage module is used to allocate the corresponding data block storage pool according to the data identifier for data blocks whose storage space does not exceed a threshold.

[0059] Furthermore, the additional judgment module is also used for:

[0060] Read the file data;

[0061] The file identifier in the file data is matched with a preset identifier, the file identifier being used to indicate the file type, and the preset identifier including an appendable identifier and a non-appendable identifier;

[0062] When the file identifier matches the preset appendable identifier, the file to be uploaded is determined to be an appendable object;

[0063] When the file identifier matches the preset non-appendable identifier, the file to be uploaded is determined to be a non-appendable object.

[0064] Furthermore, the file splitting module is also used for:

[0065] If the storage protocol of the file data is a multi-copy protocol, then the file to be uploaded will be divided into at least one data block with a storage space not exceeding the threshold.

[0066] A data block identifier is generated for each data block based on the file data identifier number and the fragment number corresponding to the data block;

[0067] Alternatively, if the storage protocol for the file data is erasure coding (EC) protocol, then the end of the file to be uploaded is padded to align with EC to obtain the target file to be uploaded;

[0068] The target file to be uploaded is split into at least one data block whose storage space does not exceed a threshold.

[0069] A data block identifier is generated for each data block based on the file data identifier number and the fragment number corresponding to the data block.

[0070] Furthermore, the data block storage module is also used for:

[0071] Obtain the identifier of the data block storage pool;

[0072] The object identifier object_ID is obtained by hashing the data block identifier using the Ceph distributed storage system and taking the modulo operation.

[0073] The data block storage pool identifier Pool_ID is obtained by hashing the identifier number of the data block storage pool through the Ceph distributed storage system.

[0074] Merge the Pool_ID and the object_ID to obtain the PG_ID;

[0075] Storage pools are allocated for each data block and file data that does not exceed the threshold, using the PG_ID.

[0076] Furthermore, the video data storage device also includes a rule loading module, which is used for:

[0077] Define CGroup hierarchical rules in the storage pool;

[0078] Determine whether the Ceph distributed storage system process is started. If the Ceph distributed storage system process is started, load the CGroup hierarchical rules and start the CGroup hierarchical rules of the Ceph distributed storage system process.

[0079] Determine whether the Ceph distributed storage system process is shut down. If the Ceph distributed storage system process is shut down, then clear the CGroup hierarchical rules.

[0080] By mounting the storage pool in the CGgroup, system resources are controlled through the CGroup to allocate resources to the Ceph distributed storage system.

[0081] Furthermore, the video data storage device also includes a restart module, which is used for:

[0082] When a restart command is received, the receiving of the file to be uploaded is stopped and uninstallation information is generated.

[0083] The radosgw mount point of the Ceph partitioned storage system is unloaded using the unloading information.

[0084] Once all the fuse mount points are unloaded, stop the file system service of the Ceph distributed storage system and generate redirection information;

[0085] The restart command is redirected to the restart service to restart the Ceph distributed storage system.

[0086] Furthermore, the video data storage device also includes a deployment module, which is used for:

[0087] Obtain the default deployment parameters of Ceph distributed storage system components. The default parameters include at least object storage gateway parameters, interface class parameters, authentication class parameters, data partitioning parameters, object storage unit parameters, and monitoring service parameters.

[0088] Obtain the deployment instructions for the Ceph distributed storage system;

[0089] Obtain the deployment instructions and deployment parameters, and generate corresponding scheduling instructions for the Ceph distributed storage system resources by combining the deployment instructions and deployment parameters in a fixed format;

[0090] The scheduling instructions are used to retrieve the software package resources and function library resources of the Ceph distributed storage system.

[0091] By selecting the binary mode transmission of the Ceph distributed storage system, the Ceph distributed storage system retrieves the software package resources and function library resources of the Ceph distributed storage system and transmits them to the target K-UX operating system;

[0092] Set the environment configuration parameters of the Ceph distributed storage system, and read the environment configuration parameters through the Ceph distributed storage system;

[0093] Deploy the environment configuration parameters to the Ceph distributed storage system;

[0094] Real-time monitoring and determination of whether the Ceph distributed storage system resources and their installation status are faulty;

[0095] If the Ceph distributed storage system resource fails, the installation of the Ceph distributed storage system resource will be reset to redeploy the Ceph distributed storage system.

[0096] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0097] A computer device includes at least one connected processor, memory, and input / output unit, wherein the memory is used to store computer-readable instructions, and the processor is used to invoke the computer-readable instructions in the memory to execute the steps of the video data storage method described above.

[0098] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0099] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the video data storage method described above.

[0100] Compared with the prior art, the embodiments of this application have the following main advantages:

[0101] When a file is written, existing metadata information is stored through the Ceph distributed storage system to quickly locate the end of the data. Data is then inserted into the current location. The storage pool protocol for the data block is obtained, and the remaining data is divided into one or more data blocks according to the storage pool protocol and stored in the backend storage. Metadata indicating the location of the written data is then saved to distribute data storage, improve read / write efficiency, and increase data reliability. Standard object storage overwrites previous data with each upload, while append-only writes overcome this constraint. By adding operation types and append positions, data is re-filled and reorganized, allowing it to be combined with historical data. Attached Figure Description

[0102] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0103] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0104] Figure 2-1 A flowchart of an embodiment of the video data storage method according to this application;

[0105] Figure 2-2 A flowchart of a multi-copy protocol for video data storage method according to this application;

[0106] Figure 2-3 A flowchart of the erasure coding protocol for the video data storage method according to this application;

[0107] Figure 3 This is a schematic diagram of a structure of an embodiment of a video data storage device according to this application;

[0108] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0109] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0110] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0111] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0112] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0113] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0114] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0115] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0116] It should be noted that the video data storage method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the video data storage device is generally located in the server / terminal device.

[0117] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0118] Referring again to Figure 2, a flowchart of an embodiment of a video data storage method according to this application is shown. The video data storage method includes the following steps:

[0119] Step 201: Obtain the file to be uploaded.

[0120] In this embodiment, an upper threshold and a lower threshold are set. If the access frequency of the file to be uploaded is higher than the upper threshold according to real-time statistics, it is judged as high-frequency data; if the access frequency of the file to be uploaded is lower than the lower threshold according to real-time statistics, it is judged as low-frequency data; if the access frequency of the file to be uploaded is between the lower threshold and the upper threshold according to real-time statistics, it is judged as medium-frequency data.

[0121] Step 202: Generate corresponding file data based on the file to be uploaded.

[0122] In this embodiment, the file data includes at least the storage pool protocol, object identifier ID, storage pool ID, bucket name, file type of the file to be uploaded, verification code, and password.

[0123] Step 203: Determine whether the file to be uploaded is an appendable object based on the file data.

[0124] In this embodiment, it is determined whether it is an append write request and the append method. If it is a normal upload mode, it is stored directly in the normal upload mode. If it is an append write process, the metadata of the object to be appended is obtained from the storage backend. This includes the position of this append (i.e., the size of the current object; append writes are uniformly required to append from the end of the object) and the rules for sharding the object in the backend storage.

[0125] Step 204: If it is determined that the file to be uploaded is an appendable object, then obtain the storage protocol of the file data and the append mode of the file to be uploaded.

[0126] In this embodiment, the append strategy includes two modes: appending at a specified position and appending without specifying a position. For the mode that appends at a specified position, it is necessary to determine whether the appended position is at the end of the file. For the mode that appends without specifying a position, appending to the end is sufficient.

[0127] Step 205: Segment the file to be uploaded according to the file data storage protocol, and generate at least one data block with a storage space not exceeding a threshold and a corresponding data block identifier.

[0128] In this embodiment, upon receiving a file to be uploaded, the server obtains the storage pool storage protocol set for the bucket by the cluster. If the file to be uploaded is determined to be an appendable object, the server obtains the storage pool storage protocol of the metadata and the append mode of the file to be uploaded. The storage pool storage protocol includes a multi-replica protocol and an erasure coding protocol, and the append mode includes appending at a specified position and appending without specifying a position.

[0129] Step 206: Create a file data storage pool and a data block storage pool.

[0130] In this embodiment, the server creates a file data storage pool and a data block storage pool. The file data storage pool is used to store metadata. The data block storage pool is used to store data blocks.

[0131] Step 207: Store the file data in the file data storage pool using the Ceph distributed storage system.

[0132] In this embodiment, when a user requests the metadata, the corresponding metadata is retrieved from the metadata storage pool and provided to the user, allowing the metadata to be stored in disk space.

[0133] Step 208: Allocate the corresponding data block storage pool according to the data blocks whose storage space does not exceed the threshold based on the data identifier.

[0134] In this embodiment, taking a live streaming room as an example, when the live streaming data needs to be sent to the user, a certain duration of data (e.g., 1 minute of video data) is placed in object storage. In a typical object storage scenario, each data upload overwrites historical data, so the live streaming platform needs to create a list and manage many files for the live streaming room to prevent file loss. Assuming the streamer streams for 6 hours, traditionally, the streamer would upload files 6 * 60 = 360 times, resulting in 360 files generated during the live stream. Over time, with a large number of streamers on the platform, this could lead to hundreds of billions of such files. The above implementation method allows for continuous uploading of files with the same filename, meaning a streamer generates only one file per live stream session, with each new file appended to the same filename. After the live stream ends, when a user needs to play a video from a specific point in time, a progress bar is used to calculate the corresponding file position, allowing the user to quickly access the file.

[0135] In this embodiment, when a file is written, existing metadata information is stored through the Ceph distributed storage system to quickly locate the end of the data. Data is then inserted into the current location. The storage pool protocol of the data block is obtained, and the remaining data is divided into one or more data blocks according to the storage pool protocol and stored in the backend storage. Then, the metadata of the data writing location information is saved to distribute data storage, improve data read / write efficiency, and increase data reliability. Standard object storage overwrites previous data with each upload, while append-only writes overcome this constraint. By adding operation types and append positions, data is re-filled and reorganized, allowing it to be combined with historical data.

[0136] In some optional implementations, the step of determining whether the file to be uploaded is an appendable object based on the file data specifically includes:

[0137] Read the file data;

[0138] The file identifier in the file data is matched with a preset identifier, the file identifier being used to indicate the file type, and the preset identifier including an appendable identifier and a non-appendable identifier;

[0139] When the file identifier matches the preset appendable identifier, the file to be uploaded is determined to be an appendable object;

[0140] When the file identifier matches the preset non-appendable identifier, the file to be uploaded is determined to be a non-appendable object.

[0141] In the above implementation, it is determined whether it is an append write request and the append method. If it is a normal upload mode, it is stored directly in the normal upload mode. If it is an append write process, the metadata of the object to be appended is obtained from the storage backend. This includes the position of this append (i.e., the size of the current object; append writes are uniformly required to append from the end of the object) and the rules for sharding the object in the backend storage.

[0142] In some optional implementations, the file data includes at least a file data identifier, and the step of segmenting the file to be uploaded according to the file data storage protocol and generating at least one data block with a storage space not exceeding a threshold and a corresponding data identifier specifically includes:

[0143] If the storage protocol of the file data is a multi-copy protocol, then the file to be uploaded will be divided into at least one data block with a storage space not exceeding the threshold.

[0144] A data block identifier is generated for each data block based on the file data identifier number and the fragment number corresponding to the data block;

[0145] Alternatively, if the storage protocol for the file data is erasure coding (EC) protocol, then the end of the file to be uploaded is padded to align with EC to obtain the target file to be uploaded;

[0146] The target file to be uploaded is split into at least one data block whose storage space does not exceed a threshold.

[0147] A data block identifier is generated for each data block based on the file data identifier number and the fragment number corresponding to the data block.

[0148] In the above implementation, the multi-replica mode works as follows: when a file is written, the tail stripe is located by storing existing metadata, and the data is filled into the current stripe. If there is data, the remaining data is split into one or more stripes according to the stripe's sharding rules and stored in the backend storage. Then, the metadata of the location information of the written data is saved for subsequent reading and appending of data. Figure 2-2 As shown, for example, if data is appended to My_object.data three times sequentially, amounting to 4MB, 12.2MB, and 7.8MB respectively, it will result in 5 stripes being stored in the backend. Erasure coding mode differs from multi-replica mode in that it requires padding with zeros at the end of each I / O operation because the EC protocol requires stripe alignment; otherwise, data overwriting will occur. Therefore, the EC redundancy ratio in the storage pool is calculated to determine the number of bytes requiring zero padding, and the trailing zeros are then stored in the backend storage. The principle of reading and writing data is the same as in multi-replica mode. For example... Figure 2-3 As shown.

[0149] In some optional implementations, the step of allocating the corresponding data block storage pool based on the data identifier for data blocks whose storage space does not exceed a threshold specifically includes:

[0150] Obtain the identifier of the data block storage pool;

[0151] The object identifier object_ID is obtained by hashing the data block identifier using the Ceph distributed storage system and taking the modulo operation.

[0152] The data block storage pool identifier Pool_ID is obtained by hashing the identifier number of the data block storage pool through the Ceph distributed storage system.

[0153] Merge the Pool_ID and the object_ID to obtain the PG_ID;

[0154] Storage pools are allocated for each data block and file data that does not exceed the threshold, using the PG_ID.

[0155] In the above implementation, for example, a storage pool is automatically generated using the command `ceph osd pool create ssd-pool 1024 1024`.

[0156] The mathematical expression for PG_ID is as follows:

[0157] Storage identifier PG_ID = hash(Pool_id) + hash(object_ID) % PG_number

[0158] If a specific data block needs to be accessed, the disk space of the storage pool containing the data block must be located. The OSD is located using the following formula:

[0159] Disk space identifier OSD_id = CRUSH(PG_ID, CLUSTER_MAP, CURSH_RULES).

[0160] In some optional implementations, after the step of obtaining the file to be uploaded, the method further includes:

[0161] Define CGroup hierarchical rules in the storage pool;

[0162] Determine whether the Ceph distributed storage system process is started. If the Ceph distributed storage system process is started, load the CGroup hierarchical rules and start the CGroup hierarchical rules of the Ceph distributed storage system process.

[0163] Determine whether the Ceph distributed storage system process is shut down. If the Ceph distributed storage system process is shut down, then clear the CGroup hierarchical rules.

[0164] By mounting the storage pool in the CGgroup, system resources are controlled through the CGroup to allocate resources to the Ceph distributed storage system.

[0165] In the above embodiments, CGroup refers to control groups, through which the allocation of physical resources, such as CPU utilization, memory usage, and disk I / O rate, can be configured;

[0166] In some optional implementations, after the steps of creating the file data storage pool and the data block storage pool, the method further includes:

[0167] When a restart command is received, the receiving of the file to be uploaded is stopped and uninstallation information is generated.

[0168] The radosgw mount point of the Ceph partitioned storage system is unloaded using the unloading information.

[0169] Once all the fuse mount points are unloaded, stop the file system service of the Ceph distributed storage system and generate redirection information;

[0170] The restart command is redirected to the restart service to restart the Ceph distributed storage system.

[0171] In the above implementation, if a problem is detected with the mount point in Ceph, the startup is stopped, multiple Ceph services are terminated, and a restart is initiated.

[0172] In some optional implementations, the step of obtaining the file to be uploaded may include:

[0173] Obtain the default deployment parameters of Ceph distributed storage system components. The default parameters include at least object storage gateway parameters, interface class parameters, authentication class parameters, data partitioning parameters, object storage unit parameters, and monitoring service parameters.

[0174] Obtain the deployment instructions for the Ceph distributed storage system;

[0175] Obtain the deployment instructions and deployment parameters, and generate corresponding scheduling instructions for the Ceph distributed storage system resources by combining the deployment instructions and deployment parameters in a fixed format;

[0176] The scheduling instructions are used to retrieve the software package resources and function library resources of the Ceph distributed storage system.

[0177] By selecting the binary mode transmission of the Ceph distributed storage system, the Ceph distributed storage system retrieves the software package resources and function library resources of the Ceph distributed storage system and transmits them to the target K-UX operating system;

[0178] Set the environment configuration parameters of the Ceph distributed storage system, and read the environment configuration parameters through the Ceph distributed storage system;

[0179] Deploy the environment configuration parameters to the Ceph distributed storage system;

[0180] Real-time monitoring and determination of whether the Ceph distributed storage system resources and their installation status are faulty;

[0181] If the Ceph distributed storage system resource fails, the installation of the Ceph distributed storage system resource will be reset to redeploy the Ceph distributed storage system.

[0182] In the above embodiments, the arrangement and deployment of environmental parameters of the Ceph distribution system were completed in the above manner.

[0183] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0184] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0185] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0186] Further reference Figure 3 As an implementation of the method shown in Figure 2 above, this application provides an embodiment of a video data storage device, which corresponds to the method embodiment shown in Figure 2, and the device can be applied to various electronic devices.

[0187] like Figure 3 As shown, the video data storage device 300 described in this embodiment includes: a file acquisition module 301, a file data generation module 302, an append judgment module 303, a protocol acquisition module 304, a file segmentation module 305, a storage pool creation module 306, a file data storage module 307, and a data block storage module 308. Wherein:

[0188] The file acquisition module 301 is used to acquire the file to be uploaded;

[0189] The file data generation module 302 is used to generate corresponding file data based on the file to be uploaded;

[0190] The append determination module 303 is used to determine whether the file to be uploaded is an appendable object based on the file data;

[0191] The protocol acquisition module 304 is used to acquire the storage protocol of the file data and the append mode of the file to be uploaded if it is determined that the file to be uploaded is an appendable object;

[0192] The file segmentation module 305 is used to segment the file to be uploaded according to the file data storage protocol, and generate at least one data block with a storage space not exceeding a threshold and a corresponding data block identifier;

[0193] Storage pool creation module 306 is used to create file data storage pools and data block storage pools;

[0194] The file data storage module 307 is used to store the file data in the file data storage pool through the Ceph distributed storage system;

[0195] The data block storage module 308 is used to allocate the corresponding data block storage pool according to the data identifier for data blocks whose storage space does not exceed a threshold.

[0196] Furthermore, the additional determination module 303 is also used for:

[0197] Read the file data;

[0198] The file identifier in the file data is matched with a preset identifier, the file identifier being used to indicate the file type, and the preset identifier including an appendable identifier and a non-appendable identifier;

[0199] When the file identifier matches the preset appendable identifier, the file to be uploaded is determined to be an appendable object;

[0200] When the file identifier matches the preset non-appendable identifier, the file to be uploaded is determined to be a non-appendable object.

[0201] Furthermore, the file splitting module 305 is also used for:

[0202] If the storage protocol of the file data is a multi-copy protocol, then the file to be uploaded will be divided into at least one data block with a storage space not exceeding the threshold.

[0203] A data block identifier is generated for each data block based on the file data identifier number and the fragment number corresponding to the data block;

[0204] Alternatively, if the storage protocol for the file data is erasure coding (EC) protocol, then the end of the file to be uploaded is padded to align with EC to obtain the target file to be uploaded;

[0205] The target file to be uploaded is split into at least one data block whose storage space does not exceed a threshold.

[0206] A data block identifier is generated for each data block based on the file data identifier number and the fragment number corresponding to the data block.

[0207] Furthermore, the data block storage module 308 is also used for:

[0208] Obtain the identifier of the data block storage pool;

[0209] The object identifier object_ID is obtained by hashing the data block identifier using the Ceph distributed storage system and taking the modulo operation.

[0210] The data block storage pool identifier Pool_ID is obtained by hashing the identifier number of the data block storage pool through the Ceph distributed storage system.

[0211] Merge the Pool_ID and the object_ID to obtain the PG_ID;

[0212] Storage pools are allocated for each data block and file data that does not exceed the threshold, using the PG_ID.

[0213] Furthermore, the video data storage device also includes a rule loading module, which is used for:

[0214] Define CGroup hierarchical rules in the storage pool;

[0215] Determine whether the Ceph distributed storage system process is started. If the Ceph distributed storage system process is started, load the CGroup hierarchical rules and start the CGroup hierarchical rules of the Ceph distributed storage system process.

[0216] Determine whether the Ceph distributed storage system process is shut down. If the Ceph distributed storage system process is shut down, then clear the CGroup hierarchical rules.

[0217] By mounting the storage pool in the CGgroup, system resources are controlled through the CGroup to allocate resources to the Ceph distributed storage system.

[0218] Furthermore, the video data storage device also includes a restart module, which is used for:

[0219] When a restart command is received, the receiving of the file to be uploaded is stopped and uninstallation information is generated.

[0220] The radosgw mount point of the Ceph partitioned storage system is unloaded using the unloading information.

[0221] Once all the fuse mount points are unloaded, stop the file system service of the Ceph distributed storage system and generate redirection information;

[0222] The restart command is redirected to the restart service to restart the Ceph distributed storage system.

[0223] Furthermore, the video data storage device also includes a deployment module, which is used for:

[0224] Obtain the default deployment parameters of Ceph distributed storage system components. The default parameters include at least object storage gateway parameters, interface class parameters, authentication class parameters, data partitioning parameters, object storage unit parameters, and monitoring service parameters.

[0225] Obtain the deployment instructions for the Ceph distributed storage system;

[0226] Obtain the deployment instructions and deployment parameters, and generate corresponding scheduling instructions for the Ceph distributed storage system resources by combining the deployment instructions and deployment parameters in a fixed format;

[0227] The scheduling instructions are used to retrieve the software package resources and function library resources of the Ceph distributed storage system.

[0228] By selecting the binary mode transmission of the Ceph distributed storage system, the Ceph distributed storage system retrieves the software package resources and function library resources of the Ceph distributed storage system and transmits them to the target K-UX operating system;

[0229] Set the environment configuration parameters of the Ceph distributed storage system, and read the environment configuration parameters through the Ceph distributed storage system;

[0230] Deploy the environment configuration parameters to the Ceph distributed storage system;

[0231] Real-time monitoring and determination of whether the Ceph distributed storage system resources and their installation status are faulty;

[0232] If the Ceph distributed storage system resource fails, the installation of the Ceph distributed storage system resource will be reset to redeploy the Ceph distributed storage system.

[0233] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0234] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0235] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0236] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 41 may include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for video data storage methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0237] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions for the video data storage method.

[0238] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0239] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the video data storage method described above.

[0240] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0241] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A video data storage method, characterized in that, Includes the following steps: Get the file to be uploaded; Generate corresponding file data based on the file to be uploaded; Determine whether the file to be uploaded is an appendable object based on the file data; If it is determined that the file to be uploaded is an appendable object, then the storage protocol of the file data and the append mode of the file to be uploaded are obtained; the append mode includes appending at a specified position and appending without specifying a position. If it is appending at a specified position, it is determined whether the specified position is the end of the file to be uploaded. The subsequent steps are only executed if the specified position is the end. If it is appending without specifying a position, it is appended to the end of the file to be uploaded by default. The file to be uploaded is segmented according to the file data storage protocol, and at least one data block with a storage space not exceeding a threshold and a corresponding data block identifier is generated. Create a file data storage pool and a data block storage pool; The file data is stored in the file data storage pool using the Ceph distributed storage system; The corresponding data block storage pool is allocated according to the data block identifier that the storage space does not exceed the threshold. The step of determining whether the file to be uploaded is an appendable object based on the file data specifically includes: Read the file data; The file identifier in the file data is matched with a preset identifier, the file identifier being used to indicate the file type, and the preset identifier including an appendable identifier and a non-appendable identifier; When the file identifier matches the appendable identifier, the file to be uploaded is determined to be an appendable object; When the file identifier matches the non-appendable identifier, the file to be uploaded is determined to be a non-appendable object; The file data includes at least a file data identifier. The step of segmenting the file to be uploaded according to the file data storage protocol and generating at least one data block with a storage space not exceeding a threshold and a corresponding data block identifier specifically includes: If the storage protocol of the file data is a multi-copy protocol, then the file to be uploaded will be divided into at least one data block with a storage space not exceeding the threshold. A data block identifier is generated for each data block based on the file data identifier number and the fragment number corresponding to the data block; Alternatively, if the storage protocol for the file data is erasure coding (EC) protocol, then the end of the file to be uploaded is padded to align with EC to obtain the target file to be uploaded; The target file to be uploaded is split into at least one data block whose storage space does not exceed a threshold. A data block identifier is generated for each data block based on the file data identifier number and the fragment number corresponding to the data block; The step of allocating the corresponding data block storage pool according to the data block identifier that the storage space does not exceed the threshold specifically includes: Obtain the identifier of the data block storage pool; The object identifier, object_ID, is obtained by hashing the data block identifier using the Ceph distributed storage system and taking the modulo operation. The data block storage pool identifier Pool_ID is obtained by hashing the identifier number of the data block storage pool through the Ceph distributed storage system. Merge the Pool_ID and the object_ID to obtain the PG_ID; Storage pools are allocated for each data block and file data that does not exceed the threshold, using the PG_ID.

2. The video data storage method according to claim 1, characterized in that, After the step of obtaining the file to be uploaded, the method further includes: Define CGroup hierarchical rules in the storage pool; Determine whether the Ceph distributed storage system process is started. If the Ceph distributed storage system process is started, load the CGroup hierarchical rules and start the CGroup hierarchical rules of the Ceph distributed storage system process. Determine whether the Ceph distributed storage system process is shut down. If the Ceph distributed storage system process is shut down, then clear the CGroup hierarchical rules. By mounting the storage pool in the CGroup, system resources are controlled through the CGroup to allocate resources to the Ceph distributed storage system.

3. The video data storage method according to claim 1, characterized in that, Following the steps of creating the file data storage pool and the data block storage pool, the method further includes: When a restart command is received, the receiving of the file to be uploaded is stopped and uninstallation information is generated. The radosgw mount point of the Ceph distributed storage system is unloaded using the unloading information. Once all fuse mount points are unloaded, stop the file system service of the Ceph distributed storage system and generate redirection information; The restart command is redirected to the restart service to restart the Ceph distributed storage system.

4. The video data storage method according to claim 1, characterized in that, Before the step of obtaining the file to be uploaded, the following steps are also included: Obtain the default deployment parameters of Ceph distributed storage system components. The default deployment parameters include at least object storage gateway parameters, interface class parameters, authentication class parameters, data partitioning parameters, object storage unit parameters, and monitoring service parameters. Obtain the deployment instructions for the Ceph distributed storage system; Obtain the deployment instructions and deployment parameters, and generate corresponding scheduling instructions for the Ceph distributed storage system resources by combining the deployment instructions and deployment parameters in a fixed format; The scheduling instructions are used to retrieve the software package resources and function library resources of the Ceph distributed storage system. By selecting the binary mode transmission of the Ceph distributed storage system, the Ceph distributed storage system retrieves the software package resources and function library resources of the Ceph distributed storage system and transmits them to the target K-UX operating system; Set the environment configuration parameters of the Ceph distributed storage system, and read the environment configuration parameters through the Ceph distributed storage system; Deploy the environment configuration parameters to the Ceph distributed storage system; Real-time monitoring and determination of whether the installation status of the software package resources and function library resources of the Ceph distributed storage system is faulty; If the software package resources or function library resources of the Ceph distributed storage system fail, the installation of the Ceph distributed storage system resources will be reset to redeploy the Ceph distributed storage system.

5. A video data storage apparatus for implementing the video data storage method as described in any one of claims 1 to 4, characterized in that, include: The file acquisition module is used to acquire the file to be uploaded; The file data generation module is used to generate corresponding file data based on the file to be uploaded. An append determination module is used to determine whether the file to be uploaded is an appendable object based on the file data; The protocol acquisition module is used to acquire the storage protocol of the file data and the append mode of the file to be uploaded if it is determined that the file to be uploaded is an appendable object. The file segmentation module is used to segment the file to be uploaded according to the file data storage protocol, and generate at least one data block with a storage space not exceeding a threshold and a corresponding data block identifier. The storage pool creation module is used to create file data storage pools and data block storage pools; The file data storage module is used to store the file data in the file data storage pool through the Ceph distributed storage system; The data block storage module is used to allocate the corresponding data block storage pool according to the data block identifier indicating that the storage space does not exceed the threshold.

6. A computer device comprising a memory and a processor, the memory storing computer-readable instructions, wherein the processor, when executing the computer-readable instructions, implements the steps of the video data storage method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the video data storage method as described in any one of claims 1 to 4.

Images