Method, device, electronic device and storage medium for accurately reading and uploading distributed file system data

Abstract

The method, device, electronic device and storage medium for accurately reading and uploading distributed file system data provided in the embodiments of the present application perform the following processing on video data of different clarity after transcoding the same real-time video stream: divide the video data into multiple small files according to a first preset time interval, merge a preset number of small files into a physical network file according to time order, and write the physical network file into the distributed file system; when performing a target small file data read operation, read the target small file from the target physical network file; when the file data is determined to be accurate based on the check value and disk drop time of the target small file, send the target small file to the video online process for online processing. The above technical solution verifies the accuracy of the file data based on the check value and disk drop time, and there is no confusion in the video content after the accurate file data is used for online processing, thereby improving the user viewing experience.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of distributed file systems, and in particular to a method, device, electronic device, and storage medium for accurately reading and uploading data of a distributed file system. Background Art

[0002] With the rapid development of cloud computing and big data technologies, the amount of global data has exploded, which has brought new development opportunities and challenges to distributed file systems. From high-performance computing to data centers, from data sharing to Internet applications, distributed file systems have penetrated into all aspects of data applications.

[0003] In a distributed file system, there is a certain time delay in the writing operation of video files. For systems with relatively low delay requirements, if a video file or part of a video file is read right after it is written, the video file has not been completely written to the distributed file system due to the above delay. As a result, there is a certain degree of inaccuracy when the file data of the video file is read from the distributed file system. As a result, when the file data is used for online processing for user viewing, the video content will be garbled, which greatly reduces the user viewing experience. Summary of the Invention

[0004] In view of this, in order to ensure that there is no confusion in the video content after online processing based on the read file data, so as to improve the user viewing experience, an embodiment of the present invention provides a method, device, electronic device and storage medium for accurately reading and uploading distributed file system data.

[0005] In a first aspect, an embodiment of the present invention provides a method for accurately reading and putting online data in a distributed file system, the method being applied to a cloud server, the method comprising:

[0006] Based on the write pod, the real-time video stream is transcoded into multiple video data with different resolutions. The write pod consists of a set of transcoding containers with different transcoding levels.

[0007] The following processing is performed for each video data with different definition: the video data is divided into multiple small files according to a first preset time interval;

[0008] Merge a preset number of small files into a physical network file based on their time sequence, and then write the physical network file to the distributed file system. The physical network file has a corresponding meta file that records the physical location of each small file in the physical network file, its checksum, and the time the small file was written to the distributed file system.

[0009] In response to a client performing a data read operation on a target small file, a target meta file is determined from a plurality of meta files based on the target small file, and the target small file is read from a target physical network file corresponding to the target meta file based on a physical location;

[0010] Determine whether the target small file's file data is accurate based on the target small file's checksum and disk write time.

[0011] When the file data of the target small file is confirmed to be accurate, the target small file is sent to the video online process for online processing.

[0012] In one possible implementation, the meta file name is composed of the main file name and the suffix .meta; wherein the main file name is obtained by rounding the Linux system timestamp / the second time interval and multiplying it by the second time interval, where the second time interval is longer than the first time interval;

[0013] The meta file includes the name, physical start address, physical end address, start frame, end frame and file number of each small file; among them, the name of each small file is based on the Linux system timestamp as the reference value, and the main file name is obtained according to the generation time of each small file and the suffix .flv.

[0014] In one possible implementation, determining a target meta file from multiple meta files based on the target small file includes:

[0015] Get the file period corresponding to the target small file; where the file period is a time period identified by the Linux system timestamp as the reference value;

[0016] The target meta file is determined based on the file period and the main file name of each meta file.

[0017] In one possible implementation, determining the target meta file based on the file period and the main file name of each meta file includes:

[0018] Determine a calculated value based on a first time and a second time interval in the file period; wherein the calculated value is obtained by rounding the first time / the second time interval and multiplying it by the second time interval, and the first time is the first time point in the file period;

[0019] Compare the calculated value with the main file name of each meta file one by one;

[0020] The meta file corresponding to the main file name that matches the comparison is determined as the target meta file.

[0021] In one possible implementation, reading the target small file from the target physical network file corresponding to the target meta file based on the physical location includes:

[0022] Check whether the target physical network file is mapped in the local memory;

[0023] When the target physical network file is not mapped in the local memory, the target physical network file is mapped from the distributed file system to the local memory through the mmap mapping technology;

[0024] Check whether the mapping record of the target small file is recorded in the target meta file;

[0025] When a mapping record is recorded in the target meta file, obtain the name of the target small file;

[0026] Read the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file;

[0027] In the case where the target physical network file is mapped in the local memory, the steps of obtaining the name of the target small file and reading the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file are executed.

[0028] In one possible implementation, the method further includes:

[0029] When the target physical network file is not mapped to the local memory but other physical network files are mapped to the local memory, the physical network file is released from the local memory.

[0030] In one possible implementation, determining whether the file data of the target small file is accurate based on the checksum and disk-down time of the target small file includes:

[0031] Determine the time difference based on the target small file's disk write time and the corresponding data read operation time;

[0032] Determine whether the time difference is less than the preset time length;

[0033] When the time difference is less than the preset time length, a CRC check is performed based on the check value to determine whether the file data of the target small file is accurate.

[0034] In one possible implementation, performing a CRC check based on the check value to determine whether the file data of the target small file is accurate includes:

[0035] Performing a checksum calculation based on the file data of the read target small file to obtain a calculated checksum;

[0036] Compare the operation check value and the check value to see if they are consistent;

[0037] When the calculated check value is consistent with the check value, it is determined that the file data of the target small file is accurate;

[0038] When the calculated check value is inconsistent with the check value, it is determined that the file data of the target small file is inaccurate.

[0039] In one possible implementation, the method further includes:

[0040] When the time difference is not less than the preset duration, the step of sending the target small file to the video online process for online processing is executed.

[0041] In one possible implementation, the method further includes:

[0042] When it is determined that the file data of the target small file is inaccurate, the target small file is read in directio mode until the CRC verification is successful, and the target small file is sent to the video online process for online processing.

[0043] In one possible implementation, sending the target small file to the video online process for online processing includes:

[0044] Save the read target small file to the local disk;

[0045] The cumulative duration of multiple target small files stored in the local disk is accumulated;

[0046] When the accumulated duration reaches a first preset duration, multiple target small files are integrated into a video clip and uploaded online.

[0047] In a second aspect, an embodiment of the present invention provides a device for accurately reading and uploading data from a distributed file system, the device being applied to a cloud server, the device comprising:

[0048] The transcoding module is used to transcode the real-time video stream into multiple video data with different resolutions based on the write pod. The write pod is composed of a set of transcoding containers with different transcoding capabilities.

[0049] The division module is configured to perform the following processing on each video data with different definition: dividing the video data into a plurality of small files according to a first preset time interval;

[0050] A writing module is used to merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file to the distributed file system; wherein, the physical network file corresponds to a meta file, which records the physical location of each small file in the physical network file, the checksum value, and the time when the small file was written to the distributed file system;

[0051] a reading module, configured to, in response to a client performing a data read operation on a target small file, determine a target meta file from a plurality of meta files based on the target small file, and read the target small file from a target physical network file corresponding to the target meta file based on a physical location;

[0052] A determination module, configured to determine whether the file data of a target small file is accurate based on the checksum value and the time the target small file was written to disk;

[0053] The online processing module is used to send the target small file to the video online process for online processing when the file data of the target small file is accurate.

[0054] In a third aspect, an embodiment of the present invention provides an electronic device, which includes: a processor and a memory, the processor is used to execute a program for accurately reading and putting online distributed file system data stored in the memory to implement the above-mentioned method for accurately reading and putting online distributed file system data.

[0055] In a fourth aspect, an embodiment of the present invention provides a storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned method of accurately reading and putting online data of a distributed file system.

[0056] Embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for accurately reading and uploading data from a distributed file system. The method includes transcoding a real-time video stream into multiple video data with different resolutions based on a write pod, wherein the write pod is composed of a group of transcoding containers with different transcoding modes. The method performs the following processing on each video data with different resolutions: dividing the video data into multiple small files at a first preset time interval; merging a preset number of small files into a physical network file based on the time sequence of the small files, and writing the physical network file to a distributed file system. The physical network file corresponds to a meta file that records the physical location, checksum, and write time of each small file in the physical network file to the distributed file system. In response to a client performing a data read operation on a target small file, a target meta file is determined from the multiple meta files based on the target small file, and a target small file is read from the target physical network file corresponding to the target meta file based on the physical location. The method also determines whether the file data of the target small file is accurate based on the checksum and write time of the target small file. If the file data of the target small file is accurate, the method sends the target small file to a video uploading process for uploading. In order to ensure the accuracy of data reading, the above technical solution verifies the accuracy of the file data in the target small file based on the verification value and the disk drop time. After the accurate file data of the target small file is used for online processing, there is no confusion in the video content, thereby improving the user viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of an embodiment of a method for accurately reading and putting data on a distributed file system provided by an embodiment of the present invention;

[0058] Figure 2 A flowchart of another embodiment of a method for accurately reading and putting data on a distributed file system provided by an embodiment of the present invention;

[0059] Figure 3 A block diagram of an embodiment of a device for accurately reading and online data from a distributed file system provided by an embodiment of the present invention;

[0060] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0062] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below with reference to the accompanying drawings. The embodiments do not limit the embodiments of the present invention.

[0063] The embodiment of the present invention provides a method for accurately reading and putting online data of a distributed file system, see Figure 1 , Figure 1 The present invention provides a flowchart of a method for accurately reading and putting data in a distributed file system. Figure 1 The process shown may include the following steps:

[0064] Step 101: transcode the real-time video stream into multiple video data with different resolutions based on the write pod;

[0065] The write pod is composed of a group of transcoding containers with different transcoding capabilities. The write pod can be considered the smallest resource management component in Kubernetes, representing the data writing process running on the cloud server. Transcoding containers include, but are not limited to, 480ppod, 720ppod, 1080ppod, 600kpod, 720HVCpod, 1080HVCpod, 4ksdrpod, 4khdrpod, and 200kpod. This group of transcoding containers can transcode the same real-time video stream, such as live video, into video data of varying resolutions to accommodate different viewing clients.

[0066] Step 102 , performing the following processing on each video data with different definition: dividing the video data into multiple small files according to a first preset time interval;

[0067] The small files herein can be understood as files storing a smaller amount of video data. The continuous video data is divided into multiple small files using a first preset time interval. For example, if the first preset time interval is 2 seconds and the video data is 30 seconds, 15 small files are obtained, each storing 2 seconds of video data. The first preset time interval is set based on actual needs and is not limited here.

[0068] Step 103: Merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file into the distributed file system;

[0069] The physical network file corresponds to a meta file (import resource file), which records the physical location of each small file in the physical network file, the checksum, and the time when the small file was written to the distributed file system.

[0070] The physical location in the meta file can be used to accurately read the required small files from the physical network file; the check value is a check value calculated based on the file data of each small file according to a specific algorithm. In this embodiment, each small file has a corresponding check value, which is used to verify whether the file data is accurate after subsequent reading; the disk placement time refers to the time when the file data in the small file is completely written to the distributed file system. This time is identified by the Linux system timestamp, where the Linux system timestamp refers to the total number of seconds from 00:00:00 on January 1, 1970 to the current time. There are two common ways to obtain the timestamp: through the time function or the gettimeofday function.

[0071] Here, the number of small files in each physical network file is limited to a certain preset number. For example, the preset number is 50. The small files store 2 seconds of video, so a physical network file is a large file that stores 100 seconds of video data. In this embodiment, by merging small files into one large file, the number of files corresponding to the video data written in the distributed file system is effectively reduced, thereby reducing the number of file access requests. For example, the number of small files corresponding to video data is 1,000. The existing technology is to write 1,000 small files into a distributed file system. When a small file needs to be accessed, the access volume is 1,000 times. When every 50 small files are merged into a physical network file, the number of physical network files is 20. The 20 physical network files are written to the distributed file system. When a small file needs to be accessed, only 20 physical network files need to be accessed. The small file can be accurately read directly from the physical network file where the small file to be accessed is located through the physical location. The access volume is reduced by 50 times. Therefore, when more than 100,000 file access requests are generated during the live broadcast, such merged processing of files can effectively reduce the number of file access requests and improve file access efficiency.

[0072] Step 104: In response to the client performing a data read operation on the target small file, a target meta file is determined from the multiple meta files based on the target small file, and the target small file is read from the target physical network file corresponding to the target meta file based on the physical location;

[0073] In specific implementation, the read operation of the target small file is implemented based on the read pod of the cloud server, that is, the read data process. In order to determine in which physical network file the target small file exists, in this embodiment, it can be determined by mapping the meta file with the correspondence between the physical network file and the small file. Therefore, it is necessary to first determine the target meta file from multiple meta files. The specific process of determining the target meta file is described in detail in the following embodiments and will not be repeated here. Since the physical location of each small file in the physical network file is recorded in the meta file, the target small file is read from the target physical network file based on the physical location of the target small file stored in the target meta file in the target physical network file.

[0074] Step 105: Determine whether the file data of the target small file is accurate based on the checksum value and the disk-down time of the target small file;

[0075] After reading the target small file, the accuracy of the file data in the target small file can be verified using the corresponding checksum and the time it was written to disk, ensuring that the subsequent generated video content is not corrupted. The specific process of determining whether the file data of the target small file is accurate is described in detail in subsequent embodiments and will not be repeated here.

[0076] Step 106: When it is determined that the file data of the target small file is accurate, the target small file is sent to the video online process for online processing.

[0077] Only when the file data is verified accurately can the target small file be sent to the video online process for processing to generate video content for users to watch. Since the file data is accurate, the generated video content does not have content confusion, which greatly improves the user viewing experience and effectively prevents user loss.

[0078] A method for accurately reading and uploading data from a distributed file system provided by an embodiment of the present invention includes transcoding a real-time video stream into multiple video data with different resolutions based on a write pod, wherein the write pod is composed of a group of transcoding containers with different transcoding modes. The following processing is performed for each video data with different resolutions: dividing the video data into multiple small files according to a first preset time interval; merging a preset number of small files into a physical network file according to the time sequence of the small files, and writing the physical network file to the distributed file system; wherein the physical network file corresponds to a meta file that records the physical location, checksum, and write time of each small file in the physical network file to the distributed file system; in response to a client performing a data read operation on a target small file, determining a target meta file from the multiple meta files based on the target small file, and reading a target small file from the target physical network file corresponding to the target meta file based on the physical location; determining whether the file data of the target small file is accurate based on the checksum and write time of the target small file; and if the file data of the target small file is accurate, sending the target small file to a video uploading process for uploading processing. In order to ensure the accuracy of data reading, the above technical solution verifies the accuracy of the file data in the target small file based on the verification value and the disk drop time. After the accurate file data of the target small file is used for online processing, there is no confusion in the video content, thereby improving the user viewing experience.

[0079] The file name of the above meta file is composed of the main file name and the suffix .meta; wherein the main file name is obtained by rounding the Linux system timestamp / the second time interval and multiplying it by the second time interval, and the second time interval is longer than the first time interval.

[0080] The second time interval can be understood as the sum of the durations of multiple small files included in a physical network file. As mentioned above, the preset number is 50, and the small files store 2s of video data. Then a physical network file is a large file that stores 100s of video data. Then the second time interval is 100s. Since the duration of the video data stored in the large file must be greater than the duration of the video data stored in the small file, the second time interval is longer than the first time interval. The above-mentioned second time interval is determined based on the number of small files included in the physical network file, and the second time interval is not limited here.

[0081] To facilitate the explanation of meta file naming, for example, the file name of the meta file is 1719565100.meta, where 1719565100 is the main file name in the meta file name. The main file name is the Linux system timestamp when the physical network file is synthesized / 100, rounded up and multiplied by 100. Since the video data included in each physical network file is 100 seconds long, the file name of the meta file corresponding to the next physical network file is 1719565200.meta, and so on.

[0082] The specific meta file includes the name, physical start address, physical end address, start frame, end frame and file number of each small file.

[0083] The name of each small file is based on the Linux system timestamp, and is composed of the main file name and the suffix .flv, which is derived from the time each small file was created. This can be understood as the Linux system timestamp corresponding to the completion of the writing process for each small file, rather than the time of the video data included in each small file, that is, the first time interval. For ease of explanation, the small files included in the meta file 1719565100.meta are used as an example, as shown below:

[0084] The first small file:

[0085] 1719565100.flv_0_193728_718647401_718647450_91045;

[0086] The second small file:

[0087] 1719565104.flv_193729_351507_718647451_718647500_91046;

[0088] The third small file:

[0089] 1719565106.flv_351508_543788_718647501_718647550_91047;

[0090] The fourth small file:

[0091] 1719565107.flv_543789_730576_718647551_718647600_91048.

[0092] Among them, 1719565100.flv is the name of the first small file, 1719565104.flv is the name of the second small file, 1719565106.flv is the name of the third small file, and 1719565107.flv is the name of the fourth small file. 1719565100 is the Linux system timestamp corresponding to when the writing process completes writing the data of the first small file, 1719565104 is the Linux system timestamp corresponding to when the writing process completes writing the data of the second small file, 1719565106 is the Linux system timestamp corresponding to when the writing process completes writing the data of the third small file, and 1719565107 is the Linux system timestamp corresponding to when the writing process completes writing the data of the fourth small file. Because the network speed affects the speed of the data writing process, the time length for the writing process to complete writing the data of a small file is inconsistent. Therefore, the time interval of the main file names of various small files is inconsistent.

[0093] 0_193728 is the physical position of the first small file in the physical network file, where 0 is the physical starting address of the first small file in the physical network file, and 193728 is the physical ending address of the first small file in the physical network file. Therefore, the physical position of a small file in the physical network file is indicated by the physical starting address and the physical starting address; 718647401 is the starting frame of the video data included in the first small file, and 718647450 is the ending frame of the video data included in the first small file. The starting frame and the ending frame can be used to indicate the starting video frame and the ending video frame of the video data specifically included in the small file; 91045 is the file number of the first small file in the multiple small files divided out. The file number can be used to check whether the small files in the physical network file are continuous, that is, the integrity of continuous small files can be checked by the file number.

[0094] Based on the understanding of meta files, in this embodiment, the process of determining the target meta file from multiple meta files based on the target small file in step 104 can be implemented by steps A1 to A2:

[0095] Step A1, obtaining the file time period corresponding to the target small file;

[0096] The file period is a time period identified by the Linux system timestamp as the reference value; it can be understood as the Linux system timestamp corresponding to the time when the video data is generated. For example, the time period of the video data is 1719565100-1719566100 using the Linux system timestamp. Each small file stores 2s of video data, so the file period corresponding to the first small file is: 1719565100-1719565102; the file period corresponding to the second small file is 1719565103-1719565104, and so on. They are not listed one by one here. In this embodiment, each small file has a corresponding file period.

[0097] In actual use, the cloud server sends the generated meta file to the technician's client for display. When the technician selects a target small file, the cloud server can obtain the file time period corresponding to the target small file.

[0098] Step A2: determining a target meta file based on the file period and the main file name of each meta file.

[0099] Specifically, the calculation value is determined based on the first time and the second time interval in the file period; the calculation value is compared with the main file name of each meta file one by one; and the meta file corresponding to the main file name that matches the comparison is determined as the target meta file.

[0100] Among them, the calculated value is the first time / second time interval rounded and multiplied by the second time interval, the first time is the first time point in the file period; the first time is the first time period in the file period, if the file period corresponding to the selected target small file is 1719565103-1719565104, then the first time is 1719565103, and the second time interval is 100 as an example, the calculated value is 1719565103 / 100 rounded and multiplied by 100 to be 1719565100, which corresponds to the main file name 1719565100 in the meta file 1719565100.meta. Therefore, the meta file 1719565100.meta is determined to be the target meta file.

[0101] The process of reading the target small file from the target physical network file corresponding to the target meta file based on the physical location in step 104 can be implemented through steps B1 to B5:

[0102] Step B1, checking whether the target physical network file is mapped in the local memory;

[0103] The above mapping can be understood as downloading the target physical network file from the distributed file system to the local memory, which is conducive to the subsequent rapid reading of the file from the local memory.

[0104] If the target physical network file is not mapped to the local memory, execute step B2; if the target physical network file is mapped to the local memory, it means that the client's reading of other target small files is also performed in the target physical network file, and it is considered that each small file in the target network file is mapped to the local memory, so execute step B4 without performing step B3.

[0105] In actual application, when the target physical network file is not mapped to the local memory but other physical network files are mapped, the physical network file is released from the local memory, effectively reducing the data occupancy of the local memory and increasing the access speed.

[0106] Step B2: Map the target physical network file from the distributed file system to the local memory using the mmap mapping technology;

[0107] The mmap mapping technology is a method of memory mapping files, which can map the target physical network file from the distributed file system to the local memory. The mapping process is an existing technology and will not be described in detail here.

[0108] Step B3: Check whether the target meta file contains a mapping record of the target small file.

[0109] During the process of mapping the physical network file to the local memory, the target small file may not have had time to be mapped to the local memory. In this case, the target small file cannot be read from the local memory. Therefore, it is necessary to determine whether the target small file is actually mapped to the local memory based on whether the mapping record of the target small file is recorded in the target meta file.

[0110] In specific implementation, the meta file also records whether each small file is mapped to the local memory. If there is a mapping record, it means that the small file has been mapped to the local memory. If there is no mapping record, it means that the small file has not been mapped to the local memory.

[0111] If the target meta file contains a mapping record, step B4 is executed. If the target meta file contains no mapping record, step B3 is continued until a mapping record is found in the target meta file.

[0112] Step B4, obtaining the name of the target small file;

[0113] When the client selects a target small file for a data read operation, the cloud server can obtain the name of the target small file selected by the client.

[0114] Step B5: Read the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file.

[0115] Compare the name of the target small file with the names of each small file in the target physical network file, and determine the small file with the same name in the target physical network file as the target small file selected by the client. Then, read the target small file from the target physical network file based on the physical location of the target small file in the target physical network file recorded in the target meta file.

[0116] exist Figure 1 Based on Figure 2 , Figure 2 A flowchart of another embodiment of a method for accurately reading and putting data in a distributed file system provided by an embodiment of the present invention. Figure 2 The process shown may include the following steps:

[0117] Step 201: transcode a real-time video stream into multiple video data with different resolutions based on a write pod; wherein the write pod is composed of a group of transcoding containers with different transcoding capabilities;

[0118] Step 202 , performing the following processing on each video data with different definition: dividing the video data into multiple small files according to a first preset time interval;

[0119] Step 203: Merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file into the distributed file system. The physical network file corresponds to a meta file, which records the physical location of each small file in the physical network file, a checksum, and the time when the small file was written to the distributed file system.

[0120] Step 204: In response to the client performing a data read operation on the target small file, a target meta file is determined from the multiple meta files based on the target small file, and the target small file is read from the target physical network file corresponding to the target meta file based on the physical location.

[0121] Step 205: determining a time difference based on the target small file's disk write time and the time corresponding to the data read operation;

[0122] The time difference is obtained by subtracting the target small file's disk flushing time from the data read operation's time.

[0123] Step 206, determining whether the time difference is less than a preset time length;

[0124] If the time difference is not less than the preset time length, it indicates that the file data in the target small file has been completely written into the distributed file system, and there is no need to perform file data verification, and execute step 208; if the time difference is less than the preset time length, the file data in the target small file may not have been completely written into the distributed file system, so it is necessary to verify the read file data to verify whether the read file data is complete and accurate, so execute step 207.

[0125] Through the above analysis, the preset time is the critical time for the file data in the small file to be completely written into the distributed file system. The above preset time is preferably set to 10 minutes. During specific implementation, the preset time is set according to actual needs and is not limited here.

[0126] Step 207: Perform a CRC check based on the check value to determine whether the file data of the target small file is accurate;

[0127] The specific process of determining whether the file data is accurate can be achieved through steps C1 to C4:

[0128] Step C1, performing a checksum calculation based on the read file data of the target small file to obtain a calculated checksum;

[0129] After removing the checksum from the read file data, the real file data stored in the target small file is obtained, and then the checksum operation is performed on the real file data according to the above-mentioned specific algorithm to obtain the calculated checksum. The specific algorithm can be modulo 2 division.

[0130] Step C2, comparing the calculated check value with the check value to see if they are consistent;

[0131] Step C3: If the calculated check value is consistent with the check value, the file data of the target small file is determined to be accurate;

[0132] When the calculated check value is consistent with the check value, it indicates that the file data in the read target small file is consistent with the file data in the target small file written in the distributed file system, and it is determined that the read file data is accurate.

[0133] Step C4: If the calculated check value is inconsistent with the check value, it is determined that the file data of the target small file is inaccurate.

[0134] If the calculated check value is inconsistent with the check value, it indicates that the file data in the read target small file is inconsistent with the file data written into the target small file in the distributed file system, and it is determined that the read file data is inaccurate.

[0135] If it is determined in step 207 that the file data of the target small file is accurate, step 208 is executed; if it is determined in step 207 that the file data of the target small file is inaccurate, step 209 is executed.

[0136] Step 208: Send the target small file to the video online process for online processing;

[0137] The specific implementation process is: storing the read target small file in the local disk; accumulating and recording the cumulative duration of multiple target small files stored in the local disk; when the cumulative duration reaches a first preset duration, integrating multiple target small files into a video clip and uploading it online.

[0138] Multiple target small files whose cumulative duration reaches a first preset duration are integrated according to the chronological order of the file periods of each target small file to form a temporally continuous video segment. The first preset duration can be set to 3 minutes, 5 minutes, or 6 minutes, etc., which is not limited here, so that each video segment of different definition is uploaded online with a duration of 3 minutes, 5 minutes, or 6 minutes, so that users can watch the video segment of each definition.

[0139] Step 209: Read the target small file in directio mode until the CRC check succeeds, and then execute step 208.

[0140] The target small file is read in directio mode. This mode can penetrate the cache and read the target small file directly from the distributed file system, but the read performance is lower. Therefore, this mode is only used to read files when the CRC check fails. The target small file is read through directio and CRC check is performed until the target small file read verification succeeds. The target small file that has passed the verification is sent to the video online process for online processing.

[0141] The embodiment of the present invention provides a device for accurately reading and putting online data of a distributed file system, which is applied to a cloud server. Figure 3 , is a block diagram of an embodiment of a device for accurately reading data from a distributed file system provided by an embodiment of the present invention. Figure 3 As shown, the device includes:

[0142] The transcoding module 301 is configured to transcode a real-time video stream into multiple video data with different resolutions based on a write pod; wherein the write pod is composed of a set of transcoding containers with different transcoding capabilities;

[0143] The division module 302 is configured to perform the following processing on each video data with different definition: dividing the video data into multiple small files according to a first preset time interval;

[0144] The writing module 303 is configured to merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file into the distributed file system. The physical network file corresponds to a meta file, which records the physical location of each small file in the physical network file, a checksum, and the time when the small file was written to the distributed file system.

[0145] The reading module 304 is configured to, in response to the client performing a data read operation on the target small file, determine a target meta file from the multiple meta files based on the target small file, and read the target small file from the target physical network file corresponding to the target meta file based on the physical location;

[0146] A determination module 305 is configured to determine whether the file data of the target small file is accurate based on the checksum value and the disk-down time of the target small file;

[0147] The online processing module 306 is used to send the target small file to the video online process for online processing when the file data of the target small file is accurate.

[0148] An embodiment of the present invention provides an apparatus for accurately reading and uploading data from a distributed file system, comprising: transcoding a real-time video stream into multiple video data with different resolutions based on a write pod; wherein the write pod is composed of a group of transcoding containers with different transcoding modes; performing the following processing for each video data with different resolutions: dividing the video data into multiple small files according to a first preset time interval; merging a preset number of small files into a physical network file according to the time sequence of the small files, and writing the physical network file to the distributed file system; wherein the physical network file corresponds to a meta file, which records the physical location, checksum, and write time of each small file in the physical network file to the distributed file system; in response to a client performing a data read operation on a target small file, determining a target meta file from the multiple meta files based on the target small file, and reading the target small file from the target physical network file corresponding to the target meta file based on the physical location; determining whether the file data of the target small file is accurate based on the checksum and write time of the target small file; and if the file data of the target small file is accurate, sending the target small file to a video uploading process for uploading processing. In order to ensure the accuracy of data reading, the above technical solution verifies the accuracy of the file data in the target small file based on the verification value and the disk drop time. After the accurate file data of the target small file is used for online processing, there is no confusion in the video content, thereby improving the user viewing experience.

[0149] Figure 4A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is provided. Figure 4 The electronic device 500 shown includes: at least one processor 501, a memory 502, at least one network interface 504 and another user interface 503. The various components in the electronic device 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 505 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 505 is not shown in FIG. Figure 4 Various buses are labeled as bus system 505.

[0150] The user interface 503 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touchpad, or a touch screen).

[0151] It is understood that the memory 502 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0152] In some embodiments, the memory 502 stores the following elements, executable units, or data structures, or a subset thereof, or an extended set thereof: an operating system 5021 and application programs 5022 .

[0153] The operating system 5021 includes various system programs, such as a framework layer, a core library layer, and a driver layer, for implementing various basic services and handling hardware-based tasks. Application programs 5022 include various application programs, such as a media player and a browser, for implementing various application services. Programs implementing the methods of the embodiments of the present invention may be included in application programs 5022.

[0154] In the embodiment of the present invention, the processor 501 is configured to execute the method steps provided in each method embodiment by calling a program or instruction stored in the memory 502 , specifically, a program or instruction stored in the application 5022 .

[0155] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 501 or by software instructions. The above processor 501 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software units in the decoding processor. The software units can be located in storage media well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 502 , and the processor 501 reads the information in the memory 502 and completes the steps of the above method in combination with its hardware.

[0156] It is understood that the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or a combination thereof.

[0157] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0158] The electronic device provided in this embodiment may be Figure 4 The electronic device shown in FIG. 1 can perform the following operations: Figure 1-2 All steps of the method for accurately reading data on the distributed file system in the online Figure 1-2 The technical effect of the method for accurately reading and uploading distributed file system data is shown in the following figure. For details, please refer to Figure 1-2 For the sake of brevity, the relevant description will not be repeated here.

[0159] An embodiment of the present invention further provides a storage medium (computer-readable storage medium). The storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and the memory may also include a combination of the aforementioned types of memory.

[0160] When one or more programs in the storage medium can be executed by one or more processors, the above-mentioned method of accurately reading and uploading data of the distributed file system can be implemented.

[0161] The processor is used to execute the program for accurately reading and putting data of the distributed file system on-line stored in the memory to implement the following steps of the method for accurately reading and putting data of the distributed file system on-line.

[0162] Professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0163] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0164] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for accurately reading data from a distributed file system, characterized in that: The method is applied to a cloud server and includes: Transcode a real-time video stream into multiple video data with different resolutions based on a write pod; wherein the write pod is composed of a set of transcoding containers with different transcoding resolutions; The following processing is performed on each of the video data with different definitions: dividing the video data into multiple small files according to a first time interval; Merging a preset number of the small files into a physical network file according to the time sequence of each of the small files, and writing the physical network file into a distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical location of each of the small files in the physical network file, a checksum, and the time when the small file was written to the distributed file system; In response to a client performing a data read operation on a target small file, determining a target meta file from the plurality of meta files based on the target small file, and reading the target small file from a target physical network file corresponding to the target meta file based on a physical location; Determining whether the file data of the target small file is accurate based on the checksum value of the target small file and the disk-writing time; When it is determined that the file data of the target small file is accurate, the target small file is sent to the video online process for online processing.

2. The method according to claim 1, characterized in that The meta file name is composed of the main file name and the suffix .meta; wherein the main file name is obtained by rounding the Linux system timestamp / the second time interval and multiplying it by the second time interval, and the second time interval is longer than the first time interval; The meta file includes the name, physical starting address, physical ending address, starting frame, ending frame and file number of each small file; wherein, the name of each small file is based on the Linux system timestamp as the reference value, and the main file name and the suffix .flv are obtained according to the generation time of each small file.

3. The method according to claim 2, characterized in that The determining a target meta file from the plurality of meta files based on the target small file includes: Obtain the file period corresponding to the target small file; wherein the file period is a time period identified by the Linux system timestamp as a reference value; A target meta file is determined based on the file period and the main file name of each meta file.

4. The method according to claim 3, characterized in that The determining of the target meta file based on the file period and the main file name of each meta file includes: Determine a calculated value based on a first time and the second time interval in the file period; wherein the calculated value is obtained by rounding the first time / the second time interval and then multiplying it by the second time interval, and the first time is the first time point in the file period; Compare the calculated value with the main file name of each meta file one by one; The meta file corresponding to the main file name that matches the comparison is determined as the target meta file.

5. The method according to claim 2, characterized in that The reading the target small file from the target physical network file corresponding to the target meta file based on the physical location includes: Check whether the target physical network file is mapped in the local memory; In a case where the target physical network file is not mapped in the local memory, mapping the target physical network file from the distributed file system to the local memory through mmap mapping technology; Check whether the target meta file contains a mapping record of the target small file; When the mapping record is recorded in the target meta file, obtaining the name of the target small file; Reading the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file; In the case where the target physical network file is mapped in the local memory, the steps of obtaining the name of the target small file and reading the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file are performed.

6. The method according to claim 5, characterized in that The method further comprises: In a case where the local memory does not map the target physical network file but has mapped other physical network files, the physical network file is released from the local memory.

7. The method according to claim 1, characterized in that The determining whether the file data of the target small file is accurate based on the checksum value of the target small file and the disk-writing time includes: Determine a time difference based on the time when the target small file is written to the disk and the time corresponding to the data read operation; Determining whether the time difference is less than a preset time length; In the case that the time difference is less than the preset time length, a CRC check is performed based on the check value to determine whether the file data of the target small file is accurate.

8. The method according to claim 7, characterized in that The performing CRC check based on the check value to determine whether the file data of the target small file is accurate includes: Performing a checksum calculation based on the read file data of the target small file to obtain a calculated checksum; Comparing the calculated check value with the check value to see if they are consistent; If the calculated check value is consistent with the check value, determining that the file data of the target small file is accurate; When the calculated check value is inconsistent with the check value, it is determined that the file data of the target small file is inaccurate.

9. The method according to claim 7, characterized in that The method further comprises: When the time difference is not less than the preset duration, the step of sending the target small file to the video online process for online processing is performed.

10. The method according to claim 7, characterized in that The method further comprises: When it is determined that the file data of the target small file is inaccurate, the target small file is read in directio mode until the CRC check is successful, and the step of sending the target small file to the video online process for online processing is executed.

11. The method according to claim 1, wherein The sending of the target small file to the video online process for online processing includes: The read target small file is stored in the local disk; Accumulate and record the cumulative duration of the plurality of target small files stored in the local disk; When the accumulated duration reaches a first preset duration, the plurality of target small files are integrated into a video clip and uploaded online.

12. A device for accurately reading data from a distributed file system, characterized in that: The device is applied to a cloud server and includes: A transcoding module, configured to transcode a real-time video stream into multiple video data with different resolutions based on a write pod; wherein the write pod is composed of a set of transcoding containers with different transcoding capabilities; a division module, configured to perform the following processing on each of the video data with different definitions: dividing the video data into a plurality of small files according to a first time interval; a writing module, configured to merge a preset number of the small files into a physical network file according to the time sequence of each of the small files, and write the physical network file into a distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical location of each of the small files in the physical network file, a checksum, and the time when the small file was written to the distributed file system; a reading module, configured to, in response to a client performing a data read operation on a target small file, determine a target meta file from the plurality of meta files based on the target small file, and read the target small file from a target physical network file corresponding to the target meta file based on a physical location; a determination module, configured to determine whether the file data of the target small file is accurate based on the checksum value of the target small file and the disk-down time; The online processing module is used to send the target small file to the video online process for online processing if the file data of the target small file is accurate.

13. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a program for accurately reading and putting data of a distributed file system on-line stored in the memory, so as to implement the method for accurately reading and putting data of a distributed file system on-line according to any one of claims 1 to 11.

14. A storage medium, characterized in that The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for accurately reading and putting online data in a distributed file system according to any one of claims 1 to 11.

Images