Method, device, electronic device and storage medium for accelerating access to small files in a distributed file system

Abstract

The method, device, electronic device and storage medium for accelerating access to small files in a distributed file system provided by the embodiments of the present application are based on writing pods to transcode a real-time video stream into multiple video data with different clarity; the following processing is performed for each video data with different clarity: the video data is divided into multiple small files according to a first preset time interval; a preset number of small files are merged and written into a physical network file according to the time sequence of each small file, and the physical network file is written into a distributed file system; the physical network file corresponds to a meta file that records the physical location of each small file in the physical network file; when performing a data read operation on the target small file, the target small file is read from the target physical network file corresponding to the target meta file based on the physical location. This strategy of merging and writing small files and performing local reading on large files effectively accelerates the access efficiency to small files.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of distributed file systems, and in particular to a method, device, electronic device, and storage medium for accelerating access to small files in a distributed file system. Background Art

[0002] With the rapid development of cloud computing and big data technologies, the amount of global data has exploded, which has brought new development opportunities and challenges to distributed file systems. From high-performance computing to data centers, from data sharing to Internet applications, distributed file systems have penetrated into all aspects of data applications.

[0003] With the development of social networks, mobile communications, and personal terminals, the proportion of small files in storage systems is increasing. When massive small files are distributed on various nodes in the network, when the client reads them, it needs to request the file access address and then access the file. Especially when processing massive data during live video recording, it is necessary to process the data in real time to generate on-demand small files. When the number of concurrent accesses to small files is large, it will cause a particularly large number of small file accesses in a short period of time, thereby slowing down the efficiency of file access. Summary of the Invention

[0004] In view of this, in order to effectively reduce the number of file access requests and improve file access efficiency, an embodiment of the present invention provides a method, device, electronic device and storage medium for accelerating access to small files in a distributed file system.

[0005] In a first aspect, an embodiment of the present invention provides a method for accelerating access to small files in a distributed file system, the method being applied to a cloud server, the method comprising:

[0006] Based on the write pod, the real-time video stream is transcoded into multiple video data with different resolutions. The write pod consists of a set of transcoding containers with different transcoding levels.

[0007] The following processing is performed for each video data with different definition: the video data is divided into multiple small files according to a first preset time interval;

[0008] Merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file into the distributed file system; wherein, the physical network file corresponds to a meta file, which records the physical location of each small file in the physical network file;

[0009] In response to the client performing a data read operation on the target small file, a target meta file is determined from multiple meta files based on the target small file, and the target small file is read from a target physical network file corresponding to the target meta file based on a physical location.

[0010] In one possible implementation, the meta file name is composed of the main file name and the suffix .meta; wherein the main file name is obtained by rounding the Linux system timestamp / the second time interval and multiplying it by the second time interval, where the second time interval is longer than the first time interval;

[0011] The meta file includes the name, physical start address, physical end address, start frame, end frame and file number of each small file; among them, the name of each small file is based on the Linux system timestamp as the reference value, and the main file name is obtained according to the generation time of each small file and the suffix .flv.

[0012] In one possible implementation, determining a target meta file from multiple meta files based on the target small file includes:

[0013] Get the file period corresponding to the target small file; where the file period is a time period identified by the Linux system timestamp as the reference value;

[0014] The target meta file is determined based on the file period and the main file name of each meta file.

[0015] In one possible implementation, determining the target meta file based on the file period and the main file name of each meta file includes:

[0016] Determine a calculated value based on a first time and a second time interval in the file period; wherein the calculated value is obtained by rounding the first time / the second time interval and multiplying it by the second time interval, and the first time is the first time point in the file period;

[0017] Compare the calculated value with the main file name of each meta file one by one;

[0018] The meta file corresponding to the main file name that matches the comparison is determined as the target meta file.

[0019] In one possible implementation, reading the target small file from the target physical network file corresponding to the target meta file based on the physical location includes:

[0020] Check whether the target physical network file is mapped in the local memory;

[0021] When the target physical network file is not mapped in the local memory, the target physical network file is mapped from the distributed file system to the local memory through the mmap mapping technology;

[0022] Check whether the mapping record of the target small file is recorded in the target meta file;

[0023] When a mapping record is recorded in the target meta file, obtain the name of the target small file;

[0024] Read the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file;

[0025] In the case where the target physical network file is mapped in the local memory, the steps of obtaining the name of the target small file and reading the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file are executed.

[0026] In one possible implementation, the method further includes:

[0027] When the local memory does not map the target physical network file but has other physical network files mapped to it, the other physical network files are released from the local memory.

[0028] In one possible implementation, the method further includes:

[0029] Save the read target small file to the local disk;

[0030] The cumulative duration of multiple target small files stored in the local disk is accumulated;

[0031] When the accumulated duration reaches the preset duration, multiple target small files are integrated into a video clip and uploaded online.

[0032] In a second aspect, an embodiment of the present invention provides a device for accelerating access to small files in a distributed file system, the device being applied to a cloud server, the device comprising:

[0033] The transcoding module is used to transcode the real-time video stream into multiple video data with different resolutions based on the write pod. The write pod is composed of a set of transcoding containers with different transcoding capabilities.

[0034] The division module is configured to perform the following processing on each video data with different definition: dividing the video data into a plurality of small files according to a first preset time interval;

[0035] A writing module is used to merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file into the distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical location of each small file in the physical network file;

[0036] The reading module is used to respond to the client performing a data read operation on the target small file, determine the target meta file from multiple meta files based on the target small file, and read the target small file from the target physical network file corresponding to the target meta file based on the physical location.

[0037] In a third aspect, an embodiment of the present invention provides an electronic device, which includes: a processor and a memory, the processor is used to execute a program for accelerating access to small files in a distributed file system stored in the memory to implement the above-mentioned method for accelerating access to small files in a distributed file system.

[0038] In a fourth aspect, an embodiment of the present invention provides a storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned method for accelerating access to small files in a distributed file system.

[0039] The invention provides a method, device, electronic device and storage medium for accelerating access to small files in a distributed file system, including: transcoding a real-time video stream into multiple video data with different resolutions based on a write pod; wherein the write pod is composed of a set of transcoding containers with different transcoding; performing the following processing for each video data with different resolutions: dividing the video data into multiple small files according to a first preset time interval; merging a preset number of small files into a physical network file according to the time sequence of each small file, and writing the physical network file into a distributed file system; wherein the physical network file corresponds to a meta file, which records the physical location of each small file in the physical network file; in response to a client performing a data read operation on a target small file, determining a target meta file from multiple meta files based on the target small file, and reading the target small file from the target physical network file corresponding to the target meta file based on the physical location. The above technical solution can merge multiple small files into a large file, namely a physical network file, and directly access the small files inside by accessing the large file, effectively reducing the number of access requests to the small files. This strategy of merging small files and performing local reads on large files effectively accelerates the access efficiency of small files. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flow chart of an embodiment of a device for accelerating access to small files in a distributed file system provided by an embodiment of the present invention;

[0041] Figure 2 A flowchart of another embodiment of a device for accelerating access to small files in a distributed file system provided by an embodiment of the present invention;

[0042] Figure 3A block diagram of an embodiment of a device for accelerating access to small files in a distributed file system provided by an embodiment of the present invention;

[0043] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0045] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below with reference to the accompanying drawings. The embodiments do not limit the embodiments of the present invention.

[0046] The embodiment of the present invention provides a method for accelerating access to small files in a distributed file system, which is applied to a cloud server. Figure 1 , Figure 1 A flowchart of an embodiment of a method for accelerating access to small files in a distributed file system provided by an embodiment of the present invention. Figure 1 The process shown may include the following steps:

[0047] Step 101: transcode the real-time video stream into multiple video data with different resolutions based on the write pod;

[0048] The write pod is composed of a group of transcoding containers with different transcoding capabilities. The write pod can be considered the smallest resource management component in Kubernetes, representing the data writing process running on the cloud server. Transcoding containers include, but are not limited to, 480ppod, 720ppod, 1080ppod, 600kpod, 720HVCpod, 1080HVCpod, 4ksdrpod, 4khdrpod, and 200kpod. This group of transcoding containers can transcode the same real-time video stream, such as live video, into video data of varying resolutions to accommodate different viewing clients.

[0049] Step 102 , performing the following processing on each video data with different definition: dividing the video data into multiple small files according to a first preset time interval;

[0050] The small files herein can be understood as files storing a smaller amount of video data. The continuous video data is divided into multiple small files using a first preset time interval. For example, if the first preset time interval is 2 seconds and the video data is 30 seconds, 15 small files are obtained, each storing 2 seconds of video data. The first preset time interval is set based on actual needs and is not limited here.

[0051] Step 103: Merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file into the distributed file system;

[0052] The physical network file corresponds to a meta file (import resource file), and the meta file records the physical location of each small file in the physical network file; therefore, the small file can be accurately read from the physical network file through the physical location.

[0053] Here, the number of small files in each physical network file is limited to a certain preset number. For example, the preset number is 50. The small files store 2 seconds of video, so a physical network file is a large file that stores 100 seconds of video data. In this embodiment, by merging small files into one large file, the number of files corresponding to the video data written in the distributed file system is effectively reduced, thereby reducing the number of file access requests. For example, the number of small files corresponding to video data is 1,000. The existing technology is to write 1,000 small files into a distributed file system. When a small file needs to be accessed, the access volume is 1,000 times. When every 50 small files are merged into a physical network file, the number of physical network files is 20. The 20 physical network files are written to the distributed file system. When a small file needs to be accessed, only 20 physical network files need to be accessed. The small file can be accurately read directly from the physical network file where the small file to be accessed is located through the physical location. The access volume is reduced by 50 times. Therefore, when more than 100,000 file access requests are generated during the live broadcast, such merged processing of files can effectively reduce the number of file access requests and improve file access efficiency.

[0054] Step 104 , in response to the client performing a data read operation on the target small file, a target meta file is determined from multiple meta files based on the target small file, and the target small file is read from a target physical network file corresponding to the target meta file based on the physical location.

[0055] In specific implementation, the read operation of the target small file is implemented based on the read pod of the cloud server, that is, the read data process. In order to determine in which physical network file the target small file exists, in this embodiment, it can be determined by mapping the meta file with the correspondence between the physical network file and the small file. Therefore, it is necessary to first determine the target meta file from multiple meta files. The specific process of determining the target meta file is described in detail in the following embodiments and will not be repeated here. Since the physical location of each small file in the physical network file is recorded in the meta file, the target small file is read from the target physical network file based on the physical location of the target small file stored in the target meta file in the target physical network file.

[0056] The embodiment of the present invention provides a method for accelerating access to small files in a distributed file system, including: transcoding a real-time video stream into multiple video data with different resolutions based on a write pod; wherein the write pod is composed of a group of transcoding containers with different transcoding; performing the following processing for each video data with different resolutions: dividing the video data into multiple small files according to a first preset time interval; merging a preset number of small files into a physical network file according to the time sequence of each small file, and writing the physical network file into a distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical location of each small file in the physical network file; in response to a client performing a data read operation on a target small file, determining a target meta file from multiple meta files based on the target small file, and reading the target small file from the target physical network file corresponding to the target meta file based on the physical location. The above technical solution can merge multiple small files into a large file, namely a physical network file, and directly access the small files inside by accessing the large file, effectively reducing the number of access requests to the small files. This strategy of merging small files and performing local reads on large files effectively accelerates the access efficiency of small files.

[0057] The file name of the above meta file is composed of the main file name and the suffix .meta; wherein the main file name is obtained by rounding the Linux system timestamp / the second time interval and multiplying it by the second time interval, and the second time interval is longer than the first time interval.

[0058] The second time interval can be understood as the sum of the durations of multiple small files included in a physical network file. As mentioned above, the preset number is 50, and the small files store 2s of video data. Then a physical network file is a large file that stores 100s of video data. Then the second time interval is 100s. Since the duration of the video data stored in the large file must be greater than the duration of the video data stored in the small file, the second time interval is longer than the first time interval. The above-mentioned second time interval is determined based on the number of small files included in the physical network file, and the second time interval is not limited here.

[0059] To explain meta file naming, consider the meta file name 1719565100.meta. 1719565100 is the main file name in the meta file. This main file name is the Linux system timestamp of the physical network file when it was synthesized, divided by 100, rounded up, and multiplied by 100. Since each physical network file contains 100 seconds of video data, the meta file corresponding to the next physical network file is named 1719565200.meta, and so on. The Linux system timestamp is the number of seconds from 00:00:00 on January 1, 1970, to the current time. There are two common ways to obtain timestamps: using the time function or the gettimeofday function.

[0060] The specific meta file includes the name, physical start address, physical end address, start frame, end frame and file number of each small file.

[0061] The name of each small file is based on the Linux system timestamp, and is composed of the main file name and the suffix .flv, which is derived from the time each small file was created. This can be understood as the Linux system timestamp corresponding to the completion of the writing process for each small file, rather than the time of the video data included in each small file, that is, the first time interval. For ease of explanation, the small files included in the meta file 1719565100.meta are used as an example, as shown below:

[0062] The first small file:

[0063] 1719565100.flv_0_193728_718647401_718647450_91045;

[0064] The second small file:

[0065] 1719565104.flv_193729_351507_718647451_718647500_91046;

[0066] The third small file:

[0067] 1719565106.flv_351508_543788_718647501_718647550_91047;

[0068] The fourth small file:

[0069] 1719565107.flv_543789_730576_718647551_718647600_91048.

[0070] Among them, 1719565100.flv is the name of the first small file, 1719565104.flv is the name of the second small file, 1719565106.flv is the name of the third small file, and 1719565107.flv is the name of the fourth small file. 1719565100 is the Linux system timestamp corresponding to when the writing process completes writing the data of the first small file, 1719565104 is the Linux system timestamp corresponding to when the writing process completes writing the data of the second small file, 1719565106 is the Linux system timestamp corresponding to when the writing process completes writing the data of the third small file, and 1719565107 is the Linux system timestamp corresponding to when the writing process completes writing the data of the fourth small file. Because the network speed affects the speed of the data writing process, the time length for the writing process to complete writing the data of a small file is inconsistent. Therefore, the time interval of the main file names of various small files is inconsistent.

[0071] 0_193728 is the physical position of the first small file in the physical network file, where 0 is the physical starting address of the first small file in the physical network file, and 193728 is the physical ending address of the first small file in the physical network file. Therefore, the physical position of a small file in the physical network file is indicated by the physical starting address and the physical starting address; 718647401 is the starting frame of the video data included in the first small file, and 718647450 is the ending frame of the video data included in the first small file. The starting frame and the ending frame can be used to indicate the starting video frame and the ending video frame of the video data specifically included in the small file; 91045 is the file number of the first small file in the multiple small files divided out. The file number can be used to check whether the small files in the physical network file are continuous, that is, the integrity of continuous small files can be checked by the file number.

[0072] Based on the understanding of meta files, in this embodiment, the process of determining the target meta file from multiple meta files based on the target small file in step 104 can be implemented by steps A1 to A2:

[0073] Step A1, obtaining the file time period corresponding to the target small file;

[0074] The file period is a time period identified by the Linux system timestamp as the reference value; it can be understood as the Linux system timestamp corresponding to the time when the video data is generated. For example, the time period of the video data is 1719565100-1719566100 using the Linux system timestamp. Each small file stores 2s of video data, so the file period corresponding to the first small file is: 1719565100-1719565102; the file period corresponding to the second small file is 1719565103-1719565104, and so on. They are not listed one by one here. In this embodiment, each small file has a corresponding file period.

[0075] In actual use, the cloud server sends the generated meta file to the technician's client for display. When the technician selects a target small file, the cloud server can obtain the file time period corresponding to the target small file.

[0076] Step A2: determining a target meta file based on the file period and the main file name of each meta file.

[0077] Specifically, the calculation value is determined based on the first time and the second time interval in the file period; the calculation value is compared with the main file name of each meta file one by one; and the meta file corresponding to the main file name that matches the comparison is determined as the target meta file.

[0078] Among them, the calculated value is the first time / second time interval rounded and multiplied by the second time interval, the first time is the first time point in the file period; the first time is the first time period in the file period, if the file period corresponding to the selected target small file is 1719565103-1719565104, then the first time is 1719565103, and the second time interval is 100 as an example, the calculated value is 1719565103 / 100 rounded and multiplied by 100 to be 1719565100, which corresponds to the main file name 1719565100 in the meta file 1719565100.meta. Therefore, the meta file 1719565100.meta is determined to be the target meta file.

[0079] exist Figure 1 Based on Figure 2 , Figure 2 A flowchart of another embodiment of a method for accelerating access to small files in a distributed file system provided by an embodiment of the present invention. Figure 2 The process shown may include the following steps:

[0080] Step 201: transcode a real-time video stream into multiple video data with different resolutions based on a write pod; wherein the write pod is composed of a group of transcoding containers with different transcoding capabilities;

[0081] Step 202 , performing the following processing on each video data with different definition: dividing the video data into multiple small files according to a first preset time interval;

[0082] Step 203: Merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file into the distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical location of each small file in the physical network file;

[0083] Step 204: In response to the client performing a data read operation on the target small file, a target meta file is determined from the multiple meta files based on the target small file, and the target physical network file is mapped in the local memory.

[0084] The above mapping can be understood as downloading the target physical network file from the distributed file system to the local memory, which is conducive to the subsequent rapid reading of the file from the local memory.

[0085] If the target physical network file is not mapped to the local memory, execute step 205; if the target physical network file is mapped to the local memory, it means that the client's reading of other target small files is also performed in the target physical network file, and it is considered that each small file in the target network file is mapped to the local memory, so execute step 206 without performing step 205.

[0086] In actual application, when the target physical network file is not mapped to the local memory but other physical network files are mapped, the physical network file is released from the local memory, effectively reducing the data occupancy of the local memory and increasing the access speed.

[0087] Step 205: Map the target physical network file from the distributed file system to the local memory using the mmap mapping technology;

[0088] The mmap mapping technology is a method of memory mapping files, which can map the target physical network file from the distributed file system to the local memory. The mapping process is an existing technology and will not be described in detail here.

[0089] Step 206: Check whether the target meta file contains a mapping record of the target small file.

[0090] During the process of mapping the physical network file to the local memory, the target small file may not have had time to be mapped to the local memory. In this case, the target small file cannot be read from the local memory. Therefore, it is necessary to determine whether the target small file is actually mapped to the local memory based on whether the mapping record of the target small file is recorded in the target meta file.

[0091] In specific implementation, the meta file also records whether each small file is mapped to the local memory. If there is a mapping record, it means that the small file has been mapped to the local memory. If there is no mapping record, it means that the small file has not been mapped to the local memory.

[0092] If the target meta file contains a mapping record, step 207 is executed. If the target meta file contains no mapping record, step 206 is executed until a mapping record is found in the target meta file.

[0093] Step 207, obtaining the name of the target small file;

[0094] When the client selects a target small file for a data read operation, the cloud server can obtain the name of the target small file selected by the client.

[0095] Step 208: Read the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file.

[0096] Compare the name of the target small file with the names of each small file in the target physical network file, and determine the small file with the same name in the target physical network file as the target small file selected by the client. Then, read the target small file from the target physical network file based on the physical location of the target small file in the target physical network file recorded in the target meta file.

[0097] In order to be able to upload video data of various clarity levels online for users to watch, in this embodiment, the read target small files are stored in the local disk; the cumulative duration of multiple target small files stored in the local disk is accumulated and recorded; when the cumulative duration reaches the preset duration, the multiple target small files are integrated into a video clip and uploaded online.

[0098] A plurality of target small files whose cumulative duration reaches a preset duration are integrated according to the time sequence of the file periods of the respective target small files to form a temporally continuous video segment.

[0099] The above-mentioned preset duration can be set to 3 minutes, 5 minutes or 6 minutes, etc., which is not limited here, so that video clips of different clarity can be uploaded online with a duration of 3 minutes, 5 minutes or 6 minutes.

[0100] The embodiment of the present invention provides a device for accelerating access to small files in a distributed file system, which is applied to a cloud server. Figure 3 , is a block diagram of an embodiment of a device for accelerating access to small files in a distributed file system provided by an embodiment of the present invention. Figure 3 As shown, the device includes:

[0101] The transcoding module 301 is configured to transcode a real-time video stream into multiple video data with different resolutions based on a write pod; wherein the write pod is composed of a set of transcoding containers with different transcoding capabilities;

[0102] The division module 302 is configured to perform the following processing on each video data with different definition: dividing the video data into multiple small files according to a first preset time interval;

[0103] A writing module 303 is configured to merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file into the distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical location of each small file in the physical network file;

[0104] The reading module 304 is used to respond to the client performing a data read operation on the target small file, determine the target meta file from multiple meta files based on the target small file, and read the target small file from the target physical network file corresponding to the target meta file based on the physical location.

[0105] The invention provides a device for accelerating access to small files in a distributed file system, comprising: transcoding a real-time video stream into multiple video data with different resolutions based on a write pod; wherein the write pod is composed of a set of transcoding containers with different transcoding; performing the following processing for each video data with different resolutions: dividing the video data into multiple small files according to a first preset time interval; merging a preset number of small files into a physical network file according to the time sequence of each small file, and writing the physical network file into a distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical location of each small file in the physical network file; in response to a client performing a data read operation on a target small file, determining a target meta file from multiple meta files based on the target small file, and reading the target small file from the target physical network file corresponding to the target meta file based on the physical location. The above technical solution can merge multiple small files into a large file, namely a physical network file, and directly access the small files inside by accessing the large file, effectively reducing the number of access requests to the small files. This strategy of merging small files and performing local reads on large files effectively accelerates the access efficiency of small files.

[0106] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is provided. Figure 4 The electronic device 500 shown includes: at least one processor 501, a memory 502, at least one network interface 504 and another user interface 503. The various components in the electronic device 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 505 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 505 is not shown in FIG. Figure 4 Various buses are labeled as bus system 505.

[0107] The user interface 503 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touchpad, or a touch screen).

[0108] It is understood that the memory 502 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0109] In some embodiments, the memory 502 stores the following elements, executable units, or data structures, or a subset thereof, or an extended set thereof: an operating system 5021 and application programs 5022 .

[0110] The operating system 5021 includes various system programs, such as a framework layer, a core library layer, and a driver layer, for implementing various basic services and handling hardware-based tasks. Application programs 5022 include various application programs, such as a media player and a browser, for implementing various application services. Programs implementing the methods of the embodiments of the present invention may be included in application programs 5022.

[0111] In an embodiment of the present invention, by calling a program or instruction stored in the memory 502, specifically, a program or instruction stored in the application 5022, the processor 501 is configured to execute the method steps provided in each method embodiment, for example, including:

[0112] Based on the write pod, the real-time video stream is transcoded into multiple video data with different resolutions. The write pod consists of a set of transcoding containers with different transcoding levels.

[0113] The following processing is performed for each video data with different definition: the video data is divided into multiple small files according to a first preset time interval;

[0114] Merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file into the distributed file system; wherein, the physical network file corresponds to a meta file, which records the physical location of each small file in the physical network file;

[0115] In response to the client performing a data read operation on the target small file, a target meta file is determined from multiple meta files based on the target small file, and the target small file is read from a target physical network file corresponding to the target meta file based on a physical location.

[0116] In one possible implementation, the meta file name is composed of the main file name and the suffix .meta; wherein the main file name is obtained by rounding the Linux system timestamp / the second time interval and multiplying it by the second time interval, where the second time interval is longer than the first time interval;

[0117] The meta file includes the name, physical start address, physical end address, start frame, end frame and file number of each small file; among them, the name of each small file is based on the Linux system timestamp as the reference value, and the main file name is obtained according to the generation time of each small file and the suffix .flv.

[0118] In one possible implementation, determining a target meta file from multiple meta files based on the target small file includes:

[0119] Get the file period corresponding to the target small file; where the file period is a time period identified by the Linux system timestamp as the reference value;

[0120] The target meta file is determined based on the file period and the main file name of each meta file.

[0121] In one possible implementation, determining the target meta file based on the file period and the main file name of each meta file includes:

[0122] Determine a calculated value based on a first time and a second time interval in the file period; wherein the calculated value is obtained by rounding the first time / the second time interval and multiplying it by the second time interval, and the first time is the first time point in the file period;

[0123] Compare the calculated value with the main file name of each meta file one by one;

[0124] The meta file corresponding to the main file name that matches the comparison is determined as the target meta file.

[0125] In one possible implementation, reading the target small file from the target physical network file corresponding to the target meta file based on the physical location includes:

[0126] Check whether the target physical network file is mapped in the local memory;

[0127] When the target physical network file is not mapped in the local memory, the target physical network file is mapped from the distributed file system to the local memory through the mmap mapping technology;

[0128] Check whether the mapping record of the target small file is recorded in the target meta file;

[0129] When a mapping record is recorded in the target meta file, obtain the name of the target small file;

[0130] Read the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file;

[0131] In the case where the target physical network file is mapped in the local memory, the steps of obtaining the name of the target small file and reading the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file are executed.

[0132] In one possible implementation, the method further includes:

[0133] When the local memory does not map the target physical network file but has other physical network files mapped to it, the other physical network files are released from the local memory.

[0134] In one possible implementation, the method further includes:

[0135] Save the read target small file to the local disk;

[0136] The cumulative duration of multiple target small files stored in the local disk is accumulated;

[0137] When the accumulated duration reaches the preset duration, multiple target small files are integrated into a video clip and uploaded online.

[0138] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 501 or by software instructions. The above processor 501 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software units in the decoding processor. The software units can be located in storage media well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 502 , and the processor 501 reads the information in the memory 502 and completes the steps of the above method in combination with its hardware.

[0139] It is understood that the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or a combination thereof.

[0140] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0141] The electronic device provided in this embodiment may be Figure 4 The electronic device shown in FIG. 1 can perform the following operations: Figure 1-2 All steps of the method for accelerating access to small files in a distributed file system, thereby achieving Figure 1-2 For details, please refer to the technical effect of the method for accelerating access to small files in the distributed file system. Figure 1-2 For the sake of brevity, the relevant description will not be repeated here.

[0142] An embodiment of the present invention further provides a storage medium (computer-readable storage medium). The storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and the memory may also include a combination of the aforementioned types of memory.

[0143] When one or more programs in the storage medium can be executed by one or more processors, the above-mentioned method for accelerating access to small files in the distributed file system can be implemented.

[0144] The processor is used to execute the program for accelerating access to small files in a distributed file system stored in the memory to implement the following steps of a method for accelerating access to small files in a distributed file system:

[0145] Based on the write pod, the real-time video stream is transcoded into multiple video data with different resolutions. The write pod consists of a set of transcoding containers with different transcoding levels.

[0146] The following processing is performed for each video data with different definition: the video data is divided into multiple small files according to a first preset time interval;

[0147] Merge a preset number of small files into a physical network file according to the time sequence of each small file, and write the physical network file into the distributed file system; wherein, the physical network file corresponds to a meta file, which records the physical location of each small file in the physical network file;

[0148] In response to the client performing a data read operation on the target small file, a target meta file is determined from multiple meta files based on the target small file, and the target small file is read from a target physical network file corresponding to the target meta file based on a physical location.

[0149] In one possible implementation, the meta file name is composed of the main file name and the suffix .meta; wherein the main file name is obtained by rounding the Linux system timestamp / the second time interval and multiplying it by the second time interval, where the second time interval is longer than the first time interval;

[0150] The meta file includes the name, physical start address, physical end address, start frame, end frame and file number of each small file; among them, the name of each small file is based on the Linux system timestamp as the reference value, and the main file name is obtained according to the generation time of each small file and the suffix .flv.

[0151] In one possible implementation, determining a target meta file from multiple meta files based on the target small file includes:

[0152] Get the file period corresponding to the target small file; where the file period is a time period identified by the Linux system timestamp as the reference value;

[0153] The target meta file is determined based on the file period and the main file name of each meta file.

[0154] In one possible implementation, determining the target meta file based on the file period and the main file name of each meta file includes:

[0155] Determine a calculated value based on a first time and a second time interval in the file period; wherein the calculated value is obtained by rounding the first time / the second time interval and multiplying it by the second time interval, and the first time is the first time point in the file period;

[0156] Compare the calculated value with the main file name of each meta file one by one;

[0157] The meta file corresponding to the main file name that matches the comparison is determined as the target meta file.

[0158] In one possible implementation, reading the target small file from the target physical network file corresponding to the target meta file based on the physical location includes:

[0159] Check whether the target physical network file is mapped in the local memory;

[0160] When the target physical network file is not mapped in the local memory, the target physical network file is mapped from the distributed file system to the local memory through the mmap mapping technology;

[0161] Check whether the mapping record of the target small file is recorded in the target meta file;

[0162] When a mapping record is recorded in the target meta file, obtain the name of the target small file;

[0163] Read the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file;

[0164] In the case where the target physical network file is mapped in the local memory, the steps of obtaining the name of the target small file and reading the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file are executed.

[0165] In one possible implementation, the method further includes:

[0166] When the local memory does not map the target physical network file but has other physical network files mapped to it, the other physical network files are released from the local memory.

[0167] In one possible implementation, the method further includes:

[0168] Save the read target small file to the local disk;

[0169] The cumulative duration of multiple target small files stored in the local disk is accumulated;

[0170] When the accumulated duration reaches the preset duration, multiple target small files are integrated into a video clip and uploaded online.

[0171] Professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0172] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0173] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for accelerating access to small files in a distributed file system, characterized in that: The method is applied to a cloud server and includes: Transcode a real-time video stream into multiple video data with different resolutions based on a write pod; wherein the write pod is composed of a set of transcoding containers with different transcoding resolutions; The following processing is performed on each of the video data with different definitions: dividing the video data into multiple small files according to a first time interval; Merging a preset number of the small files into a physical network file according to the time sequence of the small files, and writing the physical network file into a distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical location of each of the small files in the physical network file; In response to a client performing a data read operation on a target small file, determining a target meta file from the plurality of meta files based on the target small file, and reading the target small file from a target physical network file corresponding to the target meta file based on a physical location; The meta file name is composed of the main file name and the suffix .meta; wherein the main file name is obtained by dividing the Linux system timestamp by the second time interval, rounded up, and then multiplying by the second time interval; the second time interval is longer than the first time interval; The meta file includes the name, physical starting address, physical ending address, starting frame, ending frame and file number of each small file; wherein, the name of each small file is based on the Linux system timestamp as the reference value, and the main file name and the suffix .flv are obtained according to the generation time of each small file.

2. The method according to claim 1, characterized in that The determining a target meta file from the plurality of meta files based on the target small file includes: Obtain the file period corresponding to the target small file; wherein the file period is a time period identified by the Linux system timestamp as a reference value; A target meta file is determined based on the file period and the main file name of each meta file.

3. The method according to claim 2, characterized in that The determining of the target meta file based on the file period and the main file name of each meta file includes: Determine a calculated value based on a first time in the file period and the second time interval; wherein the calculated value is obtained by dividing the first time by the second time interval, rounded to an integer, and then multiplying the result by the second time interval; the first time is the first time point in the file period; Compare the calculated value with the main file name of each meta file one by one; The meta file corresponding to the main file name that matches the comparison is determined as the target meta file.

4. The method according to claim 1, wherein The reading the target small file from the target physical network file corresponding to the target meta file based on the physical location includes: Check whether the target physical network file is mapped in the local memory; In a case where the target physical network file is not mapped in the local memory, mapping the target physical network file from the distributed file system to the local memory through mmap mapping technology; Check whether the target meta file contains a mapping record of the target small file; When the mapping record is recorded in the target meta file, obtaining the name of the target small file; Reading the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file; In the case where the target physical network file is mapped in the local memory, the steps of obtaining the name of the target small file and reading the target small file from the target physical network file corresponding to the target meta file based on the name and physical location of the target small file are performed.

5. The method according to claim 4, characterized in that The method further comprises: In a case where the local memory does not map the target physical network file but has mapped other physical network files, the other physical network files are released from the local memory.

6. The method according to claim 1, wherein The method further comprises: The read target small file is stored in the local disk; Accumulate and record the cumulative duration of the plurality of target small files stored in the local disk; When the accumulated duration reaches a preset duration, the plurality of target small files are integrated into a video clip and uploaded online.

7. A device for accelerating access to small files in a distributed file system, characterized in that: The device is applied to a cloud server and includes: A transcoding module, configured to transcode a real-time video stream into multiple video data with different resolutions based on a write pod; wherein the write pod is composed of a set of transcoding containers with different transcoding capabilities; a division module, configured to perform the following processing on each of the video data with different definitions: dividing the video data into a plurality of small files according to a first time interval; a writing module, configured to merge a preset number of the small files into a physical network file according to the time sequence of the small files, and write the physical network file into a distributed file system; wherein the physical network file corresponds to a meta file, and the meta file records the physical location of each of the small files in the physical network file; a reading module, configured to, in response to a client performing a data read operation on a target small file, determine a target meta file from the plurality of meta files based on the target small file, and read the target small file from a target physical network file corresponding to the target meta file based on a physical location; The meta file name is composed of the main file name and the suffix .meta; wherein the main file name is obtained by dividing the Linux system timestamp by the second time interval, rounded up, and then multiplying by the second time interval; the second time interval is longer than the first time interval; The meta file includes the name, physical starting address, physical ending address, starting frame, ending frame and file number of each small file; wherein, the name of each small file is based on the Linux system timestamp as the reference value, and the main file name and the suffix .flv are obtained according to the generation time of each small file.

8. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a program for accelerating access to small files of a distributed file system stored in the memory, so as to implement the method for accelerating access to small files of a distributed file system according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for accelerating access to small files in a distributed file system according to any one of claims 1 to 6.

Images