Distributed file system data accurate reading and online method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a distributed file system data accurate reading and online method and device, electronic equipment and a storage medium. For video data with different definitions after the same real-time video stream is transcoded, the following processing is executed: according to a first preset time interval, the video data is divided into a plurality of small files, and according to a time sequence, a preset number of small files are merged and written into a physical network file, writing the physical network file into a distributed file system; when the target small file data reading operation is carried out, reading a target small file from the target physical network file; and sending the target small file to a video online process for online processing under the condition that the file data is determined to be accurate based on the verification value of the target small file and the disk falling time. According to the technical scheme, the accuracy of the file data is verified based on the verification value and the disk falling time, and the video content is not disordered after the accurate file data is used for online processing, so that the watching experience of a user is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of distributed file systems, and in particular to a method, device, electronic device and storage medium for accurately reading and putting online data of a distributed file system. Background Art

[0002] With the rapid development of cloud computing and big data technologies, the amount of global data has exploded, which has brought new development opportunities and challenges to distributed file systems. From high-performance computing to data centers, from data sharing to Internet applications, distributed file systems have penetrated into all aspects of data applications.

[0003] In a distributed file system, there is a certain time delay for the write operation of a video file. For a system with a relatively low delay requirement, if a video file that has just been written or part of the content of a video file is read right after it has been written, due to the above delay, the video file has not been completely written to the distributed file system. In this way, there is a certain inaccuracy when the file data of the video file is read from the distributed file system, resulting in a situation where the video content is disordered when the file data is used for online processing for users to watch, which greatly reduces the user viewing experience. Summary of the invention

[0004] In view of this, in order to ensure that there is no confusion in the video content after online processing based on the read file data, so as to improve the user viewing experience, an embodiment of the present invention provides a method, device, electronic device and storage medium for accurately reading and uploading distributed file system data.

[0005] In a first aspect, an embodiment of the present invention provides a method for accurately reading and putting online data of a distributed file system, the method being applied to a cloud server, the method comprising:

[0006] Based on the write pod, the real-time video stream is transcoded into multiple video data with different resolutions. The write pod is composed of a group of transcoding containers with different transcoding capabilities.

[0007] The following processing is performed for each video data with different definition: the video data is divided into a plurality of small files according to a first preset time interval;

[0008] Merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file into the distributed file system; wherein the physical network file corresponds to a meta file, which records the physical location of each small file in the physical network file, the check value, and the time when the small file was written to the distributed file system;

[0009] In response to the client performing a data read operation on the target small file, a target meta file is determined from the multiple meta files based on the target small file, and the target small file is read from the target physical network file corresponding to the target meta file based on the physical location;

[0010] Determine whether the file data of the target small file is accurate based on the checksum value and disk-drop time of the target small file;

[0011] When it is determined that the file data of the target small file is accurate, the target small file is sent to the video online process for online processing.

[0012] In a possible implementation, the file name of the meta file is composed of the main file name and the suffix .meta; wherein the main file name is obtained by rounding the Linux system timestamp / the second time interval and multiplying it by the second time interval, and the second time interval is longer than the first time interval;

[0013] The meta file includes the name, physical start address, physical end address, start frame, end frame and file number of each small file; among them, the name of each small file is based on the Linux system timestamp, and the main file name and the suffix .flv are obtained according to the generation time of each small file.

[0014] In a possible implementation, determining a target meta file from a plurality of meta files based on the target small file includes:

[0015] Get the file time period corresponding to the target small file; where the file time period is a time period marked with the Linux system timestamp as the reference value;

[0016] The target meta file is determined based on the file period and the main file name of each meta file.

[0017] In a possible implementation, determining the target meta file based on the file period and the main file name of each meta file includes:

[0018] Determine a calculated value based on a first time and a second time interval in the file period; wherein the calculated value is obtained by rounding the first time / the second time interval and multiplying it by the second time interval, and the first time is the first time point in the file period;

[0019] Compare the calculated value with the main file name of each meta file one by one;

[0020] The meta file corresponding to the main file name that matches the comparison is determined as the target meta file.

[0021] In a possible implementation, reading the target small file from the target physical network file corresponding to the target meta file based on the physical location includes:

[0022] Check whether the target physical network file is mapped in the local memory;

[0023] When the target physical network file is not mapped in the local memory, the target physical network file is mapped from the distributed file system to the local memory through the mmap mapping technology;

[0024] Check whether the mapping record of the target small file is recorded in the target meta file;

[0025] When a mapping record is recorded in the target meta file, obtain the name of the target small file;

[0026] Read the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file;

[0027] In the case where the local memory is mapped with the target physical network file, the steps of obtaining the name of the target small file and reading the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file are executed.

[0028] In one possible implementation, the method further includes:

[0029] When the target physical network file is not mapped in the local memory but other physical network files are mapped, the physical network file is released from the local memory.

[0030] In a possible implementation, determining whether the file data of the target small file is accurate based on the checksum value and the disk-drop time of the target small file includes:

[0031] Determine the time difference based on the time when the target small file is put into disk and the time corresponding to the data read operation;

[0032] Determine whether the time difference is less than the preset time length;

[0033] When the time difference is less than the preset time length, a CRC check is performed based on the check value to determine whether the file data of the target small file is accurate.

[0034] In a possible implementation, performing a CRC check based on the check value to determine whether the file data of the target small file is accurate includes:

[0035] Performing a checksum operation based on the file data of the read target small file to obtain a checksum operation value;

[0036] Compare the operation check value and the check value to see if they are consistent;

[0037] When the calculated check value is consistent with the check value, it is determined that the file data of the target small file is accurate;

[0038] When the calculated check value is inconsistent with the check value, it is determined that the file data of the target small file is inaccurate.

[0039] In one possible implementation, the method further includes:

[0040] When the time difference is not less than the preset duration, the step of sending the target small file to the video online process for online processing is executed.

[0041] In one possible implementation, the method further includes:

[0042] When it is determined that the file data of the target small file is inaccurate, the target small file is read in directio mode until the CRC check succeeds, and the step of sending the target small file to the video online process for online processing is executed.

[0043] In a possible implementation, sending the target small file to the video online process for online processing includes:

[0044] Save the read target small file to the local disk;

[0045] The cumulative duration of multiple target small files stored in the local disk is accumulated;

[0046] When the accumulated duration reaches a first preset duration, multiple target small files are integrated into a video clip for online upload.

[0047] In a second aspect, an embodiment of the present invention provides a device for accurately reading and putting online data of a distributed file system, the device is applied to a cloud server, and the device includes:

[0048] The transcoding module is used to transcode the real-time video stream into multiple video data with different resolutions based on the write pod; wherein the write pod is composed of a group of transcoding containers with different transcodings;

[0049] A division module is used to perform the following processing on each video data with different definition: dividing the video data into multiple small files according to a first preset time interval;

[0050] A writing module is used to merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file into the distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical location of each small file in the physical network file, the check value, and the time when the small file is written to the distributed file system;

[0051] A reading module, configured to respond to a client performing a data read operation on a target small file, determine a target meta file from a plurality of meta files based on the target small file, and read the target small file from a target physical network file corresponding to the target meta file based on a physical location;

[0052] A determination module, used to determine whether the file data of the target small file is accurate based on the checksum value and the disk-falling time of the target small file;

[0053] The online processing module is used to send the target small file to the video online process for online processing when the file data of the target small file is accurate.

[0054] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a processor and a memory, the processor being used to execute a program for accurately reading and putting online data of a distributed file system stored in the memory, so as to implement the above-mentioned method for accurately reading and putting online data of a distributed file system.

[0055] In a fourth aspect, an embodiment of the present invention provides a storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned method for accurately reading and putting online data of a distributed file system.

[0056] The method, device, electronic device and storage medium for accurately reading and uploading data of a distributed file system provided by an embodiment of the present invention include: transcoding a real-time video stream into a plurality of video data with different resolutions based on a write pod; wherein the write pod is composed of a group of transcoding containers with different transcodings; performing the following processing for each video data with different resolutions: dividing the video data into a plurality of small files according to a first preset time interval; merging a preset number of small files into a physical network file according to the time sequence of each small file, and writing the physical network file into a distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical position, check value and disk drop time of each small file in the physical network file and the small file written into the distributed file system; in response to a client performing a data read operation on a target small file, determining a target meta file from a plurality of meta files based on the target small file, and reading a target small file from a target physical network file corresponding to the target meta file based on the physical position; determining whether the file data of the target small file is accurate based on the check value and disk drop time of the target small file; and sending the target small file to a video online process for online processing when it is determined that the file data of the target small file is accurate. In order to ensure the accuracy of data reading, the above technical solution verifies the accuracy of file data in the target small file based on the verification value and the disk drop time. After online processing using the accurate file data of the target small file, there is no confusion in the video content, thereby improving the user viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of an embodiment of a method for accurately reading and putting online data of a distributed file system provided by an embodiment of the present invention;

[0058] Figure 2 A flowchart of another embodiment of a method for accurately reading and putting online data of a distributed file system provided by an embodiment of the present invention;

[0059] Figure 3 A block diagram of an embodiment of a device for accurately reading and putting online data of a distributed file system provided by an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0063] The embodiment of the present invention provides a method for accurately reading and putting online data of a distributed file system, see Figure 1 , Figure 1 A flow chart of an embodiment of a method for accurately reading and putting online data of a distributed file system provided by an embodiment of the present invention. Figure 1 The process shown may include the following steps:

[0064] Step 101, based on writing pod, transcode the real-time video stream into multiple video data with different resolutions;

[0065] The write pod is composed of a group of transcoding containers with different transcoding functions. The write pod here can be understood as the smallest resource management component in Kubernetes. The write pod represents the process of writing data running in the cloud server. The transcoding containers include but are not limited to 480ppod, 720ppod, 1080ppod, 600kpod, 720HVCpod, 1080HVCpod, 4ksdrpod, 4khdrpod, 200kpod, etc. This group of transcoding containers can transcode the same real-time video stream, such as live video, into video data of different resolutions to meet the needs of different user clients.

[0066] Step 102, performing the following processing for each video data with different definition: dividing the video data into multiple small files according to a first preset time interval;

[0067] The small file here can be understood as a file storing a small amount of video data. The first preset time interval is used to divide the above time-continuous video data into multiple small files. For example, if the first preset time interval is 2s and the video data is 30s, then 15 small files are obtained, wherein each small file stores 2s of video data. The first preset time interval is set according to actual needs and is not limited here.

[0068] Step 103, merging a preset number of small files into a physical network file according to the time sequence of each small file, and writing the physical network file into the distributed file system;

[0069] The physical network file corresponds to a meta file (import resource file), which records the physical location of each small file in the physical network file, the checksum, and the time when the small file is written to the distributed file system.

[0070] The physical location in the meta file can be used to accurately read the required small files in the physical network file; the check value is a check value calculated based on the file data of each small file according to a specific algorithm. In this embodiment, each small file has a corresponding check value, which is used to verify whether the file data is accurate after subsequent reading; the disk drop time refers to the time when the file data in the small file is completely written to the distributed file system, and the time is marked by the Linux system timestamp, where the Linux system timestamp refers to the total number of seconds from 00:00:00 on January 1, 1970 to the current time. There are two common ways to obtain the timestamp: through the time function or through the gettimeofday function.

[0071] Here, the number of small files in each physical network file is limited to a certain preset number. For example, the preset number is 50. The small files store 2s of video data, so a physical network file is a large file that stores 100s of video data. In this embodiment, by merging small files into one large file, the number of files corresponding to the video data written in the distributed file system is effectively reduced, thereby reducing the number of file access requests. For example, the number of small files corresponding to video data is 1,000. The existing technology is to write 1,000 small files into a distributed file system. When a small file needs to be accessed, the access volume is 1,000 times. When every 50 small files are merged into a physical network file, the number of physical network files is 20. The 20 physical network files are written to the distributed file system. When a small file needs to be accessed, only 20 physical network files need to be accessed. The small file can be accurately read directly from the physical network file where the small file to be accessed is located through the physical location. The access volume is reduced by 50 times. Therefore, when more than 100,000 file access requests are generated during the live broadcast, such merged file processing can effectively reduce the file access request volume and improve file access efficiency.

[0072] Step 104, in response to the client performing a data read operation on the target small file, determining a target meta file from multiple meta files based on the target small file, and reading the target small file from a target physical network file corresponding to the target meta file based on a physical location;

[0073] In the specific implementation, the read operation of the target small file is implemented based on the read pod of the cloud server, that is, the read data process. In order to determine in which physical network file the target small file exists, in this embodiment, it can be determined by a meta file that maps the correspondence between the physical network file and the small file. Therefore, it is necessary to first determine the target meta file from multiple meta files. The specific process of determining the target meta file is described in detail in the following embodiments and will not be repeated here. Since the physical location of each small file in the physical network file is recorded in the meta file, the target small file is read from the target physical network file based on the physical location of the target small file stored in the target meta file in the target physical network file.

[0074] Step 105, determining whether the file data of the target small file is accurate based on the checksum value and the disk-falling time of the target small file;

[0075] After reading the target small file, the file data in the target small file can be verified to be accurate through the check value and the time of the target small file to ensure that the subsequent generated video content is not disordered. The specific process of determining whether the file data of the target small file is accurate is described in detail in the subsequent embodiments and will not be repeated here.

[0076] Step 106, when it is determined that the file data of the target small file is accurate, the target small file is sent to the video online process for online processing.

[0077] Only when the file data is verified accurately can the target small file be sent to the video online process for processing to generate video content that users can watch. Since the file data is accurate, the generated video content does not have content confusion, which greatly improves the user viewing experience and effectively prevents user loss.

[0078] The method for accurately reading and uploading data of a distributed file system provided by an embodiment of the present invention includes: transcoding a real-time video stream into a plurality of video data with different resolutions based on a write pod; wherein the write pod is composed of a group of transcoding containers with different transcodings; performing the following processing for each video data with different resolutions: dividing the video data into a plurality of small files according to a first preset time interval; merging a preset number of small files into a physical network file according to the time sequence of each small file, and writing the physical network file into a distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical position, check value and disk drop time of each small file in the physical network file and the small file written into the distributed file system; in response to a client performing a data read operation on a target small file, determining a target meta file from a plurality of meta files based on the target small file, and reading a target small file from a target physical network file corresponding to the target meta file based on the physical position; determining whether the file data of the target small file is accurate based on the check value and disk drop time of the target small file; and sending the target small file to a video uploading process for uploading processing when it is determined that the file data of the target small file is accurate. In order to ensure the accuracy of data reading, the above technical solution verifies the accuracy of file data in the target small file based on the verification value and the disk drop time. After online processing using the accurate file data of the target small file, there is no confusion in the video content, thereby improving the user viewing experience.

[0079] The file name of the above meta file is composed of the main file name and the suffix .meta; wherein the main file name is obtained by rounding the Linux system timestamp / the second time interval and multiplying it by the second time interval, and the second time interval is longer than the first time interval.

[0080] The second time interval can be understood as the sum of the durations of multiple small files included in a physical network file. As mentioned above, the preset number is 50, and the small files store 2s of video. Then a physical network file is a large file that stores 100s of video data. In this case, the second time interval is 100s. Since the duration of the video data stored in the large file must be greater than the duration of the video data stored in the small file, the second time interval is longer than the first time interval. The above-mentioned second time interval is determined based on the number of small files included in the physical network file, and the second time interval is not limited here.

[0081] To facilitate the explanation of the naming of meta files, for example, the file name of a meta file is 1719565100.meta, where 1719565100 is the main file name in the file name of the meta file. The main file name is the Linux system timestamp / 100 when the physical network file is synthesized, rounded up and multiplied by 100. Since the duration of the video data included in each physical network file is 100s, the file name of the meta file corresponding to the next physical network file is 1719565200.meta, and so on.

[0082] The specific meta file includes the name, physical start address, physical end address, start frame, end frame and file number of each small file.

[0083] The name of each small file is based on the Linux system timestamp as the reference value, and the main file name and the suffix .flv are obtained according to the generation time of each small file. It can be understood that it is the Linux system timestamp corresponding to the completion of the writing process of each small file, rather than the time of the video data included in each small file, that is, the first time interval. For the sake of convenience, the small files included in the meta file 1719565100.meta are used as an example for explanation, as shown below:

[0084] The first small file:

[0085] 1719565100.flv_0_193728_718647401_718647450_91045;

[0086] The second small file:

[0087] 1719565104.flv_193729_351507_718647451_718647500_91046;

[0088] The third small file:

[0089] 1719565106.flv_351508_543788_718647501_718647550_91047;

[0090] The fourth small file:

[0091] 1719565107.flv_543789_730576_718647551_718647600_91048.

[0092] Among them, 1719565100.flv is the name of the first small file, 1719565104.flv is the name of the second small file, 1719565106.flv is the name of the third small file, 1719565107.flv is the name of the fourth small file, 1719565100 is the Linux system timestamp corresponding to the time when the writing process completes writing the first small file data, 1719565104 is the Linux system timestamp corresponding to the time when the writing process completes writing the second small file data, 1719565106 is the Linux system timestamp corresponding to the time when the writing process completes writing the third small file data, and 1719565107 is the Linux system timestamp corresponding to the time when the writing process completes writing the fourth small file data. Since the network speed affects the speed of the data writing process, the time length for the writing process to complete writing the data of a small file is inconsistent, so the time intervals of the main file names of various small files are inconsistent.

[0093] 0_193728 is the physical position of the first small file in the physical network file, wherein 0 is the physical starting address of the first small file in the physical network file, and 193728 is the physical ending address of the first small file in the physical network file. Therefore, the physical position of a small file in the physical network file is indicated by the physical starting address and the physical starting address; 718647401 is the starting frame of the video data included in the first small file, and 718647450 is the ending frame of the video data included in the first small file. The starting frame and the ending frame can be used to indicate the starting video frame and the ending video frame of the video data specifically included in the small file; 91045 is the file number of the first small file in the multiple small files divided. The file number can be used to check whether the small files in the physical network file are continuous, that is, the integrity of continuous small files can be checked by the file number.

[0094] Through the understanding of meta files, in this embodiment, the process of determining the target meta file from multiple meta files based on the target small file in the above step 104 can be implemented by steps A1 to A2:

[0095] Step A1, obtaining the file time period corresponding to the target small file;

[0096] The file time period is a time period identified by the Linux system timestamp as the reference value; it can be understood as the Linux system timestamp corresponding to the generation of the video data. For example, the time period of the video data is 1719565100-1719566100 using the Linux system timestamp. Each small file stores 2s of video data. Then the file time period corresponding to the first small file is: 1719565100-1719565102; the file time period corresponding to the second small file is 1719565103-1719565104, and so on. They are not listed one by one here. In this embodiment, each small file has a corresponding file time period.

[0097] In actual use, the cloud server sends the generated meta file to the technician's client for display. When the technician selects a target small file, the cloud server can obtain the file period corresponding to the target small file.

[0098] Step A2: determining a target meta file based on the file period and the main file name of each meta file.

[0099] Specifically, the calculation value is determined based on the first time and the second time interval in the file period; the calculation value is compared with the main file name of each meta file one by one; and the meta file corresponding to the main file name that matches the comparison is determined as the target meta file.

[0100] Among them, the calculated value is the first time / second time interval rounded and multiplied by the second time interval, the first time is the first time point in the file period; the first time is the first time period in the file period, such as the file period corresponding to the selected target small file is 1719565103-1719565104, then the first time is 1719565103, and the second time interval takes 100 as an example, the calculated value is 1719565103 / 100 rounded and multiplied by 100 to be 1719565100, which corresponds to the main file name 1719565100 in the meta file 1719565100.meta. Therefore, the meta file 1719565100.meta is determined to be the target meta file.

[0101] The process of reading the target small file from the target physical network file corresponding to the target meta file based on the physical location in the above step 104 can be implemented through steps B1 to B5:

[0102] Step B1, checking whether the target physical network file is mapped in the local memory;

[0103] The above mapping can be understood as downloading the target physical network file from the distributed file system to the local memory, which is conducive to the subsequent rapid reading of the file from the local memory.

[0104] When the target physical network file is not mapped to the local memory, execute step B2; when the target physical network file is mapped to the local memory, it means that the client's reading of other target small files is also performed in the target physical network file, and it is considered that each small file in the target network file is mapped to the local memory. Therefore, execute step B4 without performing step B3.

[0105] In actual application, when the target physical network file is not mapped in the local memory but other physical network files are mapped, the physical network file is released from the local memory, which effectively reduces the data occupancy of the local memory and increases the access speed.

[0106] Step B2, mapping the target physical network file from the distributed file system to the local memory through mmap mapping technology;

[0107] The mmap mapping technology is a method of memory mapping files, which can map the target physical network file from the distributed file system to the local memory. The mapping process is an existing technology and will not be described in detail here.

[0108] Step B3, querying whether the target meta file records the mapping record of the target small file;

[0109] In the process of mapping the physical network file to the local memory, the target small file may not have time to be mapped to the local memory, so the target small file cannot be read from the local memory. Therefore, it is necessary to determine whether the target small file is actually mapped to the local memory based on whether the mapping record of the target small file is recorded in the target meta file.

[0110] In specific implementation, the meta file also records whether each small file is mapped to the local memory. If there is a mapping record, it means that the small file has been mapped to the local memory. If there is no mapping record, it means that the small file has not been mapped to the local memory.

[0111] If the target meta file has a mapping record, step B4 is executed. If the target meta file has no mapping record, step B3 is continued until a mapping record is found in the target meta file.

[0112] Step B4, obtaining the name of the target small file;

[0113] When the client selects a target small file for a data read operation, the cloud server can obtain the name of the target small file selected by the client.

[0114] Step B5, reading the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file.

[0115] The name of the target small file is compared with the names of each small file in the target physical network file, and the small file with the same name in the target physical network file is determined as the target small file selected by the client. Then, the target small file is read from the target physical network file based on the physical location of the target small file in the target physical network file recorded in the target meta file.

[0116] exist Figure 1 Based on Figure 2 , Figure 2 A flowchart of another embodiment of a method for accurately reading and putting data on a distributed file system provided by an embodiment of the present invention. Figure 2 The process shown may include the following steps:

[0117] Step 201, based on a write pod, transcode a real-time video stream into a plurality of video data with different resolutions; wherein the write pod is composed of a group of transcoding containers with different transcodings;

[0118] Step 202, performing the following processing for each video data with different definition: dividing the video data into multiple small files according to a first preset time interval;

[0119] Step 203: merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file into the distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical location of each small file in the physical network file, the check value, and the time when the small file is written to the distributed file system;

[0120] Step 204, in response to the client performing a data read operation on the target small file, determining a target meta file from multiple meta files based on the target small file, and reading the target small file from a target physical network file corresponding to the target meta file based on a physical location;

[0121] Step 205, determining the time difference based on the time when the target small file is put into the disk and the time corresponding to the data reading operation;

[0122] The time difference is obtained by subtracting the time when the target small file is written to the disk from the time corresponding to the data read operation.

[0123] Step 206, determining whether the time difference is less than a preset time length;

[0124] If the time difference is not less than the preset time length, it indicates that the file data in the target small file has been completely written into the distributed file system, and there is no need to verify the file data, and execute step 208; if the time difference is less than the preset time length, the file data in the target small file may not have been completely written into the distributed file system, so it is necessary to verify the read file data to verify whether the read file data is complete and accurate, so execute step 207.

[0125] Through the above analysis, the preset time length is the critical time length for the file data in the small file to be completely written into the distributed file system. The above preset time length is preferably set to 10 minutes. During specific implementation, the preset time length is set according to actual needs and is not limited here.

[0126] Step 207, performing a CRC check based on the check value to determine whether the file data of the target small file is accurate;

[0127] The specific process of determining whether the file data is accurate can be achieved by steps C1 to C4:

[0128] Step C1, performing a check value calculation based on the file data of the read target small file to obtain a calculated check value;

[0129] After removing the check value from the read file data, the real file data stored in the target small file is obtained, and then the check value operation is performed on the real file data according to the above-mentioned specific algorithm to obtain the operation check value. The specific algorithm can be modulo 2 division.

[0130] Step C2, comparing the calculated check value and the check value to see if they are consistent;

[0131] Step C3, when the calculated check value is consistent with the check value, determining that the file data of the target small file is accurate;

[0132] When the calculated check value is consistent with the check value, it means that the file data in the read target small file is consistent with the file data in the target small file written in the distributed file system, and it is determined that the read file data is accurate.

[0133] Step C4: when the calculated check value is inconsistent with the check value, it is determined that the file data of the target small file is inaccurate.

[0134] When the calculated check value is inconsistent with the check value, it indicates that the file data in the read target small file is inconsistent with the file data in the target small file written in the distributed file system, and it is determined that the read file data is inaccurate.

[0135] If it is determined in step 207 that the file data of the target small file is accurate, step 208 is executed; if it is determined in step 207 that the file data of the target small file is inaccurate, step 209 is executed.

[0136] Step 208, sending the target small file to the video online process for online processing;

[0137] The specific implementation process is: storing the read target small file in the local disk; accumulating and recording the cumulative duration of multiple target small files stored in the local disk; when the cumulative duration reaches a first preset duration, integrating the multiple target small files into a video clip for online upload.

[0138] The multiple target small files whose cumulative duration reaches the first preset duration are integrated according to the time sequence of the file time periods of each target small file to integrate into a time-continuous video segment. The above-mentioned first preset duration can be set to a duration of 3 minutes, 5 minutes or 6 minutes, etc., which is not limited here, so that each video segment of different definition is put online for 3 minutes, 5 minutes or 6 minutes, so that users can watch video segments of various definitions.

[0139] Step 209, read the target small file in directio mode until the CRC check succeeds, and execute step 208.

[0140] The directio mode is used to read the target small file. Since this mode can penetrate the cache and directly read the target small file from the distributed file system, but the reading performance is low, this mode is used to read the file only when the above CRC check fails. The target small file is read through directio and the CRC check is performed until the target small file read check succeeds. The target small file that succeeds the check is sent to the video online process for online processing.

[0141] The embodiment of the present invention provides a device for accurately reading and putting online data of a distributed file system, and the device is applied to a cloud server, see Figure 3 , is a block diagram of an embodiment of a device for accurately reading and putting online data of a distributed file system provided by an embodiment of the present invention. Figure 3 As shown, the device comprises:

[0142] The transcoding module 301 is used to transcode the real-time video stream into multiple video data with different resolutions based on the write pod; wherein the write pod is composed of a group of transcoding containers with different transcodings;

[0143] The division module 302 is used to perform the following processing on each video data with different definition: divide the video data into multiple small files according to a first preset time interval;

[0144] The writing module 303 is used to merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file into the distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical location of each small file in the physical network file, the check value, and the time when the small file is written to the distributed file system;

[0145] The reading module 304 is used for determining a target meta file from a plurality of meta files based on the target small file in response to the client performing a data read operation on the target small file, and reading the target small file from a target physical network file corresponding to the target meta file based on a physical location;

[0146] A determination module 305, configured to determine whether the file data of the target small file is accurate based on the checksum value and the disk-falling time of the target small file;

[0147] The online processing module 306 is used to send the target small file to the video online process for online processing when the file data of the target small file is accurate.

[0148] The device for accurately reading and uploading data of a distributed file system provided by an embodiment of the present invention comprises: transcoding a real-time video stream into a plurality of video data with different resolutions based on a write pod; wherein the write pod is composed of a group of transcoding containers with different transcodings; performing the following processing for each video data with different resolutions: dividing the video data into a plurality of small files according to a first preset time interval; merging a preset number of small files into a physical network file according to the time sequence of each small file, and writing the physical network file into a distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical position, check value and disk drop time of each small file in the physical network file and the small file written into the distributed file system; in response to a client performing a data read operation on a target small file, determining a target meta file from a plurality of meta files based on the target small file, and reading a target small file from a target physical network file corresponding to the target meta file based on the physical position; determining whether the file data of the target small file is accurate based on the check value and disk drop time of the target small file; and sending the target small file to a video online process for online processing when it is determined that the file data of the target small file is accurate. In order to ensure the accuracy of data reading, the above technical solution verifies the accuracy of file data in the target small file based on the verification value and the disk drop time. After online processing using the accurate file data of the target small file, there is no confusion in the video content, thereby improving the user viewing experience.

[0149] Figure 4A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, Figure 4 The electronic device 500 shown includes: at least one processor 501, a memory 502, at least one network interface 504 and other user interfaces 503. The various components in the electronic device 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 505 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 505 is not shown in FIG. Figure 4 Various buses are labeled as bus system 505 .

[0150] The user interface 503 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0151] It can be understood that the memory 502 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0152] In some implementations, the memory 502 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 5021 and an application program 5022 .

[0153] The operating system 5021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 5022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 5022.

[0154] In the embodiment of the present invention, by calling the program or instruction stored in the memory 502, specifically, the program or instruction stored in the application 5022, the processor 501 is used to execute the method steps provided by each method embodiment.

[0155] The method disclosed in the above embodiment of the present invention can be applied to the processor 501, or implemented by the processor 501. The processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 501 or the instruction in the form of software. The above processor 501 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 502, and the processor 501 reads the information in the memory 502 and completes the steps of the above method in combination with its hardware.

[0156] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0157] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0158] The electronic device provided in this embodiment may be Figure 4 The electronic device shown in FIG. 1 may perform the following steps: Figure 1-2 All steps of the method for accurately reading and putting data on the distributed file system in the Figure 1-2 The technical effect of the method for accurately reading and putting online distributed file system data is shown in the figure. For details, please refer to Figure 1-2 For the sake of brevity, the relevant description is not repeated here.

[0159] The embodiment of the present invention also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0160] When one or more programs in the storage medium can be executed by one or more processors, the above-mentioned method of accurately reading and putting online data of the distributed file system can be implemented.

[0161] The processor is used to execute a program for accurately reading and putting data on a distributed file system stored in a memory, so as to implement the following steps of a method for accurately reading and putting data on a distributed file system.

[0162] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0163] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0164] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for accurately reading and putting online data of a distributed file system, characterized in that: The method is applied to a cloud server, and the method comprises: Based on a write pod, a real-time video stream is transcoded into a plurality of video data with different resolutions; wherein the write pod is composed of a group of transcoding containers with different transcodings; The following processing is performed for each video data with different definition: the video data is divided into a plurality of small files according to a first preset time interval; Merge a preset number of the small files into a physical network file according to the time sequence of each of the small files, and write the physical network file into a distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical location of each of the small files in the physical network file, a check value, and the time when the small files are written to the distributed file system; In response to the client performing a data read operation on a target small file, a target meta file is determined from the multiple meta files based on the target small file, and the target small file is read from a target physical network file corresponding to the target meta file based on a physical location; Determining whether the file data of the target small file is accurate based on the check value of the target small file and the disk-dropping time; When it is determined that the file data of the target small file is accurate, the target small file is sent to the video online process for online processing.

2. The method according to claim 1, characterized in that The file name of the meta file is composed of the main file name and the suffix .meta; wherein the main file name is obtained by rounding the Linux system timestamp / the second time interval multiplied by the second time interval, and the second time interval is longer than the first time interval; The meta file includes the name, physical start address, physical end address, start frame, end frame and file number of each small file; wherein, the name of each small file is based on the Linux system timestamp as the reference value, and the main file name and the suffix .flv are obtained according to the generation time of each small file.

3. The method according to claim 2, characterized in that The determining a target meta file from the plurality of meta files based on the target small file includes: Obtain the file time period corresponding to the target small file; wherein the file time period is a time period identified by using the Linux system timestamp as a reference value; A target meta file is determined based on the file period and the main file name of each of the meta files.

4. The method according to claim 3, characterized in that: The determining the target meta file based on the file period and the main file name of each meta file includes: Determine a calculated value based on a first time in the file period and the second time interval; wherein the calculated value is obtained by rounding the first time / the second time interval by the second time interval, and the first time is the first time point in the file period; Compare the calculated value with the main file name of each meta file one by one; The meta file corresponding to the main file name that matches the comparison is determined as the target meta file.

5. The method according to claim 2, characterized in that: The reading the target small file from the target physical network file corresponding to the target meta file based on the physical location includes: Check whether the target physical network file is mapped in the local memory; In the case where the target physical network file is not mapped in the local memory, mapping the target physical network file from the distributed file system to the local memory through mmap mapping technology; Query whether the target small file mapping record is recorded in the target meta file; In the case where the mapping record is recorded in the target meta file, obtaining the name of the target small file; Reading the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file; In the case where the local memory is mapped with the target physical network file, the steps of obtaining the name of the target small file and reading the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file are performed.

6. The method according to claim 5, characterized in that The method further comprises: When the local memory does not map the target physical network file but has mapped other physical network files, the physical network file is released from the local memory.

7. The method according to claim 1, characterized in that The determining whether the file data of the target small file is accurate based on the check value of the target small file and the disk-falling time includes: Determine the time difference based on the time when the target small file is put into the disk and the time corresponding to the data reading operation; Determine whether the time difference is less than a preset time length; In the case where the time difference is less than the preset time length, a CRC check is performed based on the check value to determine whether the file data of the target small file is accurate.

8. The method according to claim 7, characterized in that The performing CRC check based on the check value to determine whether the file data of the target small file is accurate includes: Performing a check value calculation based on the read file data of the target small file to obtain a calculation check value; Comparing the calculated check value with the check value to see if they are consistent; In the case where the calculated check value is consistent with the check value, determining that the file data of the target small file is accurate; When the calculated check value is inconsistent with the check value, it is determined that the file data of the target small file is inaccurate.

9. The method according to claim 7, characterized in that: The method further comprises: When the time difference is not less than the preset duration, the step of sending the target small file to the video online process for online processing is performed.

10. The method according to claim 7, characterized in that The method further comprises: When it is determined that the file data of the target small file is inaccurate, the target small file is read in directio mode until the CRC check succeeds, and the step of sending the target small file to the video online process for online processing is executed.

11. The method according to claim 1, characterized in that: The step of sending the target small file to the video online process for online processing includes: The read target small file is stored in the local disk; Accumulate and record the accumulated duration of the plurality of target small files stored in the local disk; When the accumulated duration reaches a first preset duration, the plurality of target small files are integrated into a video clip for uploading.

12. A device for accurately reading data from a distributed file system, characterized in that: The device is applied to a cloud server, and the device includes: A transcoding module, used for transcoding a real-time video stream into a plurality of video data with different resolutions based on a write pod; wherein the write pod is composed of a group of transcoding containers with different transcodings; A division module, configured to perform the following processing on each video data with different definition: dividing the video data into a plurality of small files according to a first preset time interval; A writing module, used to merge a preset number of the small files into a physical network file according to the time sequence of each of the small files, and write the physical network file into a distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical location of each of the small files in the physical network file, the check value, and the time when the small files are written to the distributed file system; A reading module, configured to, in response to a client performing a data read operation on a target small file, determine a target meta file from the plurality of meta files based on the target small file, and read the target small file from a target physical network file corresponding to the target meta file based on a physical location; A determination module, configured to determine whether the file data of the target small file is accurate based on the check value of the target small file and the disk-falling time; The online processing module is used to send the target small file to the video online process for online processing if the file data of the target small file is accurate.

13. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a program for accurately reading and putting online data of a distributed file system stored in the memory, so as to implement the method for accurately reading and putting online data of a distributed file system according to any one of claims 1 to 11.

14. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for accurately reading and putting online data of a distributed file system according to any one of claims 1 to 11.

Images