File merging method, device, equipment, storage medium, product and system

Abstract

The invention provides a file merging method, device and equipment, a storage medium, a product and a system, and the method comprises the steps: dividing a plurality of to-be-merged files into a plurality of to-be-merged file groups according to a preset grouping strategy; calling a plurality of sub-threads to respectively execute a first task to obtain a plurality of task execution results, the first task including merging the to-be-merged files in one to-be-merged file group according to the time sequence of the to-be-merged files to generate a first merged file, and executing the first merged file according to the time sequence of the to-be-merged files; the task execution result comprises the first combined file and a time sequence attribute of the first combined file; and according to the time sequence attributes of the first combined files, combining the plurality of first combined files to generate a second combined file. According to the method and the device, the time sequence of the data can be kept in the merging process, the consistency and the accuracy of the merged files on the time sequence are ensured, the time required by file merging can be greatly shortened, and the overall processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of file merging, and in particular to a file merging method, device, equipment, storage medium, product and system. Background Art

[0002] In many scenarios of data processing and file management, file merging is a basic and critical processing task. With the rapid development of information technology, various types of data are constantly generated and accumulated in the form of files, and the amount of data has increased dramatically. How to efficiently manage and utilize this data has become an urgent problem to be solved. Especially when processing a large number of files with time-series attributes (such as log files, transaction records, etc.), how to quickly and accurately merge these files to ensure the integrity and timeliness of the data is particularly important. Summary of the invention

[0003] In view of this, the embodiments of the present application are committed to providing a file merging method, device, equipment, storage medium, product and system, which can not only maintain the time order of data during the merging process and ensure the consistency and accuracy of the merged files in terms of time sequence, but also significantly shorten the time required for file merging and improve overall processing efficiency.

[0004] According to a first aspect of an embodiment of the present application, a file merging method is provided, comprising:

[0005] According to a preset grouping strategy, the plurality of files to be merged are divided into a plurality of file groups to be merged, wherein the grouping strategy includes a strategy for grouping based on the time sequence of the files to be merged;

[0006] Calling multiple sub-threads to respectively execute a first task to obtain multiple task execution results, wherein the first task includes merging the files to be merged in a group of files to be merged according to the time sequence of the files to be merged to generate a first merged file, and the task execution result includes the first merged file and the time sequence attribute of the first merged file, and each sub-thread executes at least one first task;

[0007] According to the time sequence attribute of the first merged file, multiple first merged files are merged to generate a second merged file.

[0008] Optionally, dividing the plurality of files to be merged into a plurality of file groups to be merged according to a preset grouping strategy includes:

[0009] Determine the number of target groups based on currently available processing resource information;

[0010] According to the time sequence attribute of each file to be merged, and in accordance with the time balance distribution principle and the same group distribution principle of adjacent time ranges, the multiple files to be merged are divided into the target grouping number of file groups to be merged.

[0011] Optionally, the first task also includes:

[0012] According to a preset division rule, the first merged file is logically divided into a plurality of content blocks;

[0013] According to the time sequence attribute of each content block in the first merged file, an index of each content block is established, wherein the index includes a time index;

[0014] Generate an index file of the first merged file according to the index of each content block;

[0015] The task execution result also includes an index file of the first merged file;

[0016] The method further comprises:

[0017] The index files of the first merged files are merged to generate an index file of the second merged file.

[0018] Optionally, the step of establishing an index for each content block includes:

[0019] Establishing an offset index of each content block according to the offset of each content block in the first merged file;

[0020] A time index of the offset index is established according to a time sequence attribute of the first merged file.

[0021] Optionally, the method further includes:

[0022] In response to a query request, dividing the query task corresponding to the query request into at least one query subtask according to a preset task division strategy, wherein the query request includes a query task for querying target data in a first file group, the first file group includes the second merged file, and the query subtask includes querying the target data in a second merged file;

[0023] Execute the at least one query subtask in parallel to obtain a query result corresponding to each query subtask;

[0024] The query results corresponding to each subtask are merged using the preset merging rules to generate the final query result.

[0025] According to a second aspect of an embodiment of the present application, a file merging device is provided, including:

[0026] The first unit is used to divide the plurality of to-be-merged files into a plurality of to-be-merged file groups according to a preset grouping strategy, wherein the grouping strategy includes a strategy for grouping based on the time sequence of the to-be-merged files;

[0027] The second unit is used to call multiple sub-threads to respectively execute the first task to obtain multiple task execution results, wherein the first task includes merging the files to be merged in a group of files to be merged according to the time sequence of the files to be merged to generate a first merged file, and the task execution result includes the first merged file and the time sequence attribute of the first merged file, and each sub-thread executes at least one first task;

[0028] The third unit is used to merge multiple first merged files according to the time sequence attribute of the first merged files to generate a second merged file.

[0029] According to a third aspect of an embodiment of the present application, there is provided a file merging device, including a memory and a processor;

[0030] The memory is connected to the processor and is used to store programs;

[0031] The processor is used to implement the file merging method as described in any one of the first aspects of the embodiments of the present application by running the program in the memory.

[0032] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the file merging method as described in any one of the first aspects of the embodiments of the present application is implemented.

[0033] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, enable the processor to implement a file merging method as described in any one of the first aspects of the embodiments of the present application.

[0034] According to a sixth aspect of an embodiment of the present application, a file merging system is provided, including:

[0035] The file merging device and the data storage module as described in the third aspect of the embodiment of the present application, wherein the data storage module is communicatively connected with the file merging device;

[0036] The data storage module is used to store the second merged file generated by the file merging device.

[0037] The file merging method proposed in the present application divides multiple files to be merged into multiple file groups to be merged through a preset grouping strategy, and the files in each group are organized according to certain timing attributes. Subsequently, the file groups to be merged are assigned to different sub-threads, and each sub-thread is responsible for merging all files in at least one file group to generate a first merged file with clear timing attributes. Finally, according to the timing attributes of the first merged file, the first merged file is finally merged to generate a second merged file. On the one hand, the present application can maintain the time sequence of data during the merging process, ensure the consistency and accuracy of the merged files in terms of timing, and facilitate the execution of downstream tasks; on the other hand, it can significantly shorten the time required for file merging and improve the overall processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0039] Figure 1 A schematic diagram of the structure of a file merging system provided in an embodiment of the present application.

[0040] Figure 2 A flowchart of the first file merging method provided in an embodiment of the present application.

[0041] Figure 3 A flowchart of the second file merging method provided in an embodiment of the present application.

[0042] Figure 4 A flowchart of the third file merging method provided in an embodiment of the present application.

[0043] Figure 5 A schematic diagram of the structure of a file merging device provided in an embodiment of the present application.

[0044] Figure 6 A schematic diagram of the structure of a file merging device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solution of the embodiment of the present application is applicable to the application processing of large-scale data file merging scenarios with time series attributes, such as manufacturing, scientific research, autonomous driving and intelligent transportation, aerospace, and Internet of Things monitoring and other multi-application scenarios. In these application scenarios, it is usually necessary to merge a large number of scattered small files with time series relationships into a few or a large file to facilitate subsequent data analysis, visualization, and storage management. For example, in the manufacturing industry, the merged analysis of equipment operation logs is crucial for fault prediction and maintenance plan formulation; in the field of scientific research, the merging of large-scale experimental data helps to quickly verify and deeply analyze research results; in the field of Internet of Things monitoring, the merging of massive time series data generated by monitoring equipment is of great significance for anomaly detection and system optimization. Exemplarily, in a cloud control platform based on vehicle-road collaboration, trajectory file merging is one of the core application scenarios, which is crucial for achieving efficient data management and utilization. The cloud control system needs to process more than 10,000 vehicle trajectory data, which is reported at a frequency of seconds, requiring the platform to have the storage capacity of TB-level massive data. Specifically, a trajectory file is generated every second, and up to 86,400 trajectory files can be accumulated within a day. This amount of data undoubtedly brings about the problem of massive small files. In order to perform effective data analysis, visualization, and storage management, these scattered trajectory files need to be merged into a few large files. The technical solution of the embodiment of the present application can maintain the time sequence of the data during the merging process, ensure the consistency and accuracy of the merged files in terms of time sequence, facilitate the execution of downstream tasks, and significantly shorten the time required for file merging, thereby improving overall processing efficiency.

[0046] The technical solution provided in the embodiments of the present application can be exemplarily applied to hardware devices such as processors, electronic devices, servers (including cloud servers), or packaged into software programs to be run. When the hardware device executes the processing of the technical solution of the embodiments of the present application, or the above-mentioned software program is run, the automatic division of target tasks and the automatic calling of the application program interface required for the task can be achieved to complete the purpose of the target task. The embodiments of the present application only exemplarily introduce the specific processing of the technical solution of the present application, and do not limit the specific implementation form of the technical solution of the present application. Any technical implementation form that can execute the processing of the technical solution of the present application can be adopted by the embodiments of the present application.

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] Before introducing this application solution, the relevant technologies are first introduced:

[0049] In many scenarios of data processing and file management, file merging is a basic and critical processing task. With the rapid development of information technology, various types of data are constantly generated and accumulated in the form of files, and the amount of data has increased dramatically. How to efficiently manage and utilize this data has become an urgent problem to be solved. Especially when processing a large number of files with time-series attributes (such as log files, transaction records, etc.), how to quickly and accurately merge these files to ensure the integrity and timeliness of the data is particularly important.

[0050] Traditional file merging methods often use a single thread to execute sequentially. This method may be effective enough when processing small-scale data sets, but when faced with massive data, its inefficiency and time-consuming shortcomings are exposed. In addition, when processing files with time-series attributes, traditional methods are difficult to ensure that the merged files strictly follow the original time sequence, which affects the accuracy and availability of the data to a certain extent.

[0051] In view of this, the embodiments of the present application provide a file merging method, device, equipment, storage medium, product and system, which can maintain the time order of data during the merging process, ensure the consistency and accuracy of the merged files in time, facilitate the execution of downstream tasks, and significantly shorten the time required for file merging, thereby improving overall processing efficiency. They are described in detail one by one in the following embodiments.

[0052] Exemplary Systems

[0053] For ease of understanding, the implementation environment of the file merging method provided in the embodiment of the present application is first introduced exemplarily. Figure 1 , Figure 1 This is a structural schematic diagram of a file merging system provided in an embodiment of the present application. The file merging method provided in the present application can be exemplarily applied to the file merging system.

[0054] like Figure 1 As shown, the file merging system includes a file merging device 110 and a file storage module 120. The file merging device 110 and the file storage module 120 are in communication connection.

[0055] The file merging device 110 is used to receive files to be merged sent by various methods such as upstream task transmission or user upload, and execute the file merging method provided by the embodiment of the present application, including: first, according to a preset grouping strategy, multiple files to be merged are divided into multiple file groups to be merged, and the grouping strategy includes a strategy for grouping based on the timing of the files to be merged; then multiple sub-threads are called to respectively execute the first task to obtain multiple task execution results, the first task includes merging the files to be merged in a file group to be merged according to the timing of the files to be merged to generate a first merged file, the task execution result includes the first merged file and the timing attribute of the first merged file, and each sub-thread executes at least one first task; finally, according to the timing attribute of the first merged file, multiple first merged files are merged to generate a second merged file. After the second merged file is generated, the generated second merged file is stored in the file storage module 120.

[0056] As for the specific content of the file merging method, it will be explained in subsequent embodiments. Please refer to the content of the subsequent embodiments and will not be described in detail here.

[0057] Optionally, the file merging system further includes a first input module (not shown in the figure), which is respectively connected to the file merging device 110 and the file storage module 120. The first input module is used to receive the files to be merged transmitted by the upstream task or input by the user, and temporarily store the received files to be merged in the first submodule of the file storage module 120. When the preset file merging task execution conditions are met, the file merging device 110 obtains the temporarily stored files to be merged that need to be merged from the file storage module 120, executes the file merging method provided in the embodiment of the present application, generates a second merged file corresponding to the files to be merged, and stores the second merged file in the second submodule of the file storage module 120.

[0058] Optionally, after the second merged file storage value file is stored in the second submodule, the corresponding file to be merged stored in the first submodule is deleted. Exemplarily, the original file to be merged is compressed, the compressed file is named with YYYYMMDDHH as the file prefix, and then the compressed file is moved to the backup directory for archiving, and finally the original file to be merged is cleared.

[0059] Optionally, the file merging device also includes a second input module (not shown in the figure), a data query module (not shown in the figure) and an output module (not shown in the figure), wherein the data query module is communicatively connected to the second input module, the output module and the data storage module respectively.

[0060] The second input module is used to receive query request information and send the query request information to the data query module, wherein the query request information includes a query task for querying target data in a first file group, and the first file group includes a plurality of the second merged files.

[0061] The data query module is used to divide the query task corresponding to the query request information into at least one query subtask according to a preset task division strategy after receiving the query request information, process the at least one subtask in parallel, obtain the query result corresponding to each query subtask, and merge the query results corresponding to each subtask using a preset merging rule to generate a final query result, and finally send the final query result to the output module. Wherein, the query subtask includes querying the target data in a second merged file.

[0062] The output module is used to output the final query result.

[0063] It should be noted that there is a time sequence relationship between the multiple files to be merged in the above application scenario, which will be explained in subsequent content and will not be explained in detail.

[0064] The above-mentioned communication connection can be a wireless communication connection or a wired communication connection or any combination thereof, and can be a direct communication connection or an indirect communication connection or any combination thereof, which is not limited in the present application.

[0065] The above-mentioned file merging device 110 can be various types of servers, which can be an independent server, a server group composed of multiple physical servers, or a cloud server providing cloud computing services, etc. The specific type and setting method of the file merging device 110 are determined according to the application scenario and performance requirements, and this application does not limit this.

[0066] Exemplary Methods

[0067] The file merging method provided in the embodiment of the present application can be exemplarily applied to Figure 1 The file merging system in the file merging system is specifically executed by the file merging device in the file merging system.

[0068] Figure 2 The following is a flow chart of the first file merging method provided in the embodiment of the present application. Figure 2 As shown, the method includes steps S201-S203:

[0069] S201. Divide a plurality of to-be-merged files into a plurality of to-be-merged file groups according to a preset grouping strategy, wherein the grouping strategy includes a strategy for grouping the to-be-merged files based on a time sequence.

[0070] The files to be merged can be understood as files that need to be merged, and these files may have multiple sources, such as trajectory files, log files, video clips, audio clips, data reports or any other files that need to be integrated into a more complete and continuous file type.

[0071] In addition, the files to be merged have a timing attribute, which can be understood as the timestamp or time-related information attached to the internal data of the file or the file itself, and is used to represent the temporal order of the file, such as the file creation time, the file modification time, the file reception time, or the time reflected by the file content, etc. For example, a vehicle trajectory file will record the generation time of the file, a log file will usually record the time when an event occurs, and a video or audio file may also contain recording time information, etc. According to the timing attributes of the files to be merged, the timing of the files to be merged can be determined, that is, the temporal order relationship between the files to be merged. Exemplarily, the files to be merged are vehicle trajectory files, and the timing of the files to be merged is the time order in which the vehicle trajectory files are generated.

[0072] The to-be-merged file group includes a plurality of to-be-merged files.

[0073] Before executing step S201, it is indeed necessary to determine whether a file merging task needs to be executed. A trigger condition or trigger mechanism is preset, and when it is determined that the preset trigger condition is met, the file merging method is determined to be executed and step S201 is executed.

[0074] Optionally, a periodic file merging method can be set up, which automatically triggers the execution of the file merging method at certain time intervals (such as every hour, every day or every week). It is suitable for data that needs to be regularly sorted and processed, such as vehicle trajectory files, log files, transaction records, etc. Optionally, when the number of files to be merged reaches a preset number threshold, the system automatically triggers the execution of the file merging method. The number threshold can be set according to the processing capacity and storage limit of the system. Optionally, a total occupancy threshold of the files to be merged is set. When the total occupancy of the files to be merged reaches the threshold, the system automatically triggers the execution of the file merging method.

[0075] The grouping strategy can be understood as a method for grouping the files to be merged according to certain rules and conditions before the file merging is performed. The grouping strategy is preset and stored in the cloud or device side. When the file merging device determines that a file merging task needs to be performed, it calls the preset grouping strategy and divides the multiple files to be merged into multiple file groups to be merged according to the grouping strategy.

[0076] The grouping strategy includes a strategy for grouping based on the time sequence of the files to be merged.

[0077] When the files to be merged have time series attributes, we can sort the files to be merged according to these time series information, and divide the files to be merged into different groups according to certain rules (such as the continuity of time periods, the proximity of time points, etc.), to ensure that in application scenarios that need to maintain time continuity (such as trajectory file merging, video clip merging, log file merging, etc.), there is a certain continuity or correlation in time, so as to ensure that in the subsequent merging process, the files can be merged step by step in chronological order to avoid time confusion.

[0078] As an optional implementation, step S201 "dividing the multiple files to be merged into multiple file groups to be merged according to a preset grouping strategy" includes steps A1-A2:

[0079] A1. Determine the target group quantity based on currently available processing resource information.

[0080] The currently available processing resource information may be understood as the system hardware resources available at the current moment, such as the number of CPU cores.

[0081] Optionally, the number of sub-threads that can be called is determined according to the currently available processing resource information, and then the number of target groups is determined according to the number of sub-threads that can be called.

[0082] Ideally, the number of groups should match the number of child threads to maximize parallel processing efficiency.

[0083] Exemplarily, the number of CPU cores of the merged server is 32 cores, 1.2 is used as the empirical coefficient, and a rounding-down strategy is adopted to determine that the number of sub-threads that can be called is 38. In order to maximize the parallel processing efficiency, the target number of groups is also determined to be 38.

[0084] A2. According to the time sequence attribute of each file to be merged, the multiple files to be merged are divided into the target number of file groups to be merged according to the time balance distribution principle and the same group distribution principle of adjacent time ranges.

[0085] The time-balanced allocation rule can be understood as evenly dividing the total time range covered by the files to be allocated according to the number of target groups, and then evenly allocating the generation merged files to each file group to be allocated in chronological order according to the time range corresponding to each file group to be allocated. For example, it is set to automatically trigger the execution of the file merging method every first time (for example, one hour), and determine the number of target groups to be the first number (for example, ten groups). According to the time-balanced allocation strategy, the time period that the files to be merged in each group should cover is the first time divided by the first number (that is, each hour is divided into ten segments, each segment is six minutes).

[0086] The principle of allocating adjacent time ranges to the same group can be understood as files in the same to-be-merged file group are all files with adjacent time ranges.

[0087] As an optional implementation, if some files are particularly large or small, or the number of files in a certain period of time is particularly large or small, then simple time balancing may cause some threads to have too heavy or too light workloads. In this case, load balancing allocation rules are used instead of time balancing allocation rules to divide the files to be merged.

[0088] The load balancing distribution rule can be understood as dynamically distributing files to various groups based on the size of the files to be merged and the processing power of the worker threads to ensure that the workload of each worker thread or processing core is roughly equal. This rule can dynamically adjust the grouping based on the size and number of files to achieve a more balanced workload distribution.

[0089] S202, calling multiple sub-threads to respectively execute the first task to obtain multiple task execution results, wherein the first task includes merging the files to be merged in a group of files to be merged according to the sequence of the files to be merged to generate a first merged file, and the task execution result includes the first merged file and the sequence attribute of the first merged file, and each sub-thread executes at least one first task.

[0090] The first task is to merge the files in a group of files to be merged according to the time sequence of the files to be merged to generate the first merged file. This is a subtask in the file merging process and is executed by the subthread.

[0091] The task execution result refers to the result obtained after the child thread executes the first task, including the generated first merged file and the timing attributes of the first merged file.

[0092] The timing attribute is used to represent the timestamp or time-related information attached to the file internal data of the first merged file or the file itself, such as the start time of the first merged file (i.e., the start time of the earliest file in the group) and the end time (i.e., the end time of the latest file in the group). The timing attribute of the first merged file can be used to determine the timing of multiple first files to be merged, i.e., the temporal order of the first merged files, so as to maintain the correct timing in the subsequent merging process.

[0093] Optionally, in order to efficiently manage child threads, thread pool technology can be used. The thread pool can create and manage a group of child threads in advance. When there is a first task to be executed, according to the preset task allocation strategy, the corresponding number of child threads are obtained from the thread pool to execute the first task. After the task is completed, the first thread will not be destroyed immediately, but will return to the thread pool to wait for the next task.

[0094] Furthermore, the task allocation principle may be to use an average allocation strategy according to the number of file groups to be merged and the number of available sub-threads to evenly allocate the file groups to be merged to each sub-thread.

[0095] Optionally, the task allocation strategy may adopt a dynamic allocation strategy to dynamically adjust task allocation according to the real-time load conditions of the child threads to achieve load balancing.

[0096] Preferably, the number of first tasks is equal to the number of sub-threads, and each sub-thread executes one first task respectively, so as to shorten the processing time to the maximum extent.

[0097] Furthermore, when executing the first task, the child thread needs to perform the merging operation according to the time sequence of the files to be merged.

[0098] Optionally, the files to be merged are read one by one in chronological order, and a merge operation is performed to ensure the chronological correctness of the first merged file.

[0099] Optionally, the merging operation may be performed in an appending manner to ensure the local orderliness of the first merged file, that is, each merged sub-file is stored in order in ascending order of timestamps.

[0100] In the process of executing the first task, the subthread adopts a parallel processing method to ensure that multiple groups of files to be merged can be processed at the same time, thereby improving the overall efficiency of file merging. And because the files to be merged are grouped according to the time sequence of the files to be merged, the files to be merged are divided into multiple groups of files to be merged, and for each group of files to be merged, the files to be merged in the group are merged according to the time sequence of the files to be merged in the group, so the first merged file generated after the files in each group of files to be merged also has a time sequence attribute, and there is a time sequence relationship between the multiple files to be merged.

[0101] Specifically, first, the main thread creates a corresponding number of first tasks according to the number of file groups to be merged, and allocates these tasks to child threads for execution according to a preset allocation principle, and the allocation can be static allocation (that is, each child thread processes a fixed number of tasks) or dynamic allocation (that is, tasks are dynamically allocated according to the completion status of the child thread and the system load); then the child thread starts to execute the first task. When executing the first task, the child thread will merge the files to be merged in the file group to be merged according to the timing of the files to be merged to generate a first merged file. During the merging process, the child thread needs to record and update the timing attributes of the first merged file; when the child thread completes the first task, it will return the generated first merged file and its timing attributes to the main thread as the task execution result; the main thread obtains multiple first merged files and their timing attributes.

[0102] S203: Merge multiple first merged files according to the time sequence attributes of the first merged files to generate a second merged file.

[0103] The first merged file is a file generated by the child thread in step S202. Each first merged file represents a merge result of all files in a to-be-merged file group and has its own specific time sequence attribute.

[0104] Specifically, after step S202 is completed, the main thread collects the first merged files corresponding to all the file groups to be merged and their time sequence attributes, and then merges the multiple first merged files according to the time sequence attributes of the first merged files using the file merging technology to obtain the final second merged file. The second merged file contains the contents of all the original files to be merged and is arranged in the correct time sequence.

[0105] Optionally, the second merged file is output to a specified location or storage medium.

[0106] Optionally, merging multiple first merged files according to the time sequence attribute of the first merged files may include: sorting all the first merged files according to the time sequence attribute of each first merged file, creating a new empty file, and appending the contents of each first merged file to the created empty file one by one in the sorted order to generate a final second merged file. In this process, the continuity and integrity of the file content need to be ensured.

[0107] Through the above implementation, step S203 can ensure that all first merged files are merged into a complete second merged file in the correct time sequence, thereby completing the entire file merging process.

[0108] Optionally, during the file merging process in step S202 and / or step S203, if merged files with overlapping timing (i.e., the timing attributes of two or more merged files overlap) are found during the sorting process, it is necessary to decide how to handle these overlapping parts based on specific business needs. For example, you can choose to retain the earliest file content, the latest file content, or perform some form of merging.

[0109] Optionally, during the file merging process in step S202 and / or step S203, an error handling mechanism is implemented to deal with possible problems, such as insufficient disk space, file read and write permission problems, etc.

[0110] Optionally, during the file merging process in step S202 and / or step S203, a user interface or log information is provided so that the user can understand the progress and result of the merging process.

[0111] The file merging method proposed in the present application divides multiple files to be merged into multiple file groups to be merged through a preset grouping strategy, and the files in each group are organized according to certain timing attributes. Subsequently, the file groups to be merged are assigned to different sub-threads, and each sub-thread is responsible for merging all files in at least one file group to generate a first merged file with clear timing attributes. Finally, according to the timing attributes of the first merged file, the first merged file is finally merged to generate a second merged file. On the one hand, the present application can maintain the time sequence of data during the merging process, ensure the consistency and accuracy of the merged files in terms of timing, and facilitate the execution of downstream tasks; on the other hand, it can significantly shorten the time required for file merging and improve the overall processing efficiency.

[0112] Figure 3 The following is a flow chart of the second file merging method provided in the embodiment of the present application. As an optional implementation method, Figure 3 As shown, the file merging method provided in this embodiment also includes steps S301-S304:

[0113] In this embodiment, the sub-thread executing the first task further includes steps S301-S303:

[0114] S301. Logically divide the first merged file into multiple content blocks according to a preset division rule.

[0115] The content blocks can be understood as a plurality of parts or paragraphs into which the first merged file is logically divided. Each content block can contain a certain amount of data or information, and all content blocks together constitute the entire first merged file. Optionally, a content block corresponds to the content of an original file to be merged.

[0116] The division rules may be based on file size, time interval, specific mark, etc., and may be set according to actual conditions to ensure that the size and quantity of content blocks are suitable for index creation and subsequent content access. This application does not limit this.

[0117] Optionally, the first merged file is logically divided into a plurality of content blocks at fixed time intervals, and the content of each content block covers a time range. Since the first merged file contains a time sequence attribute, the divided content blocks also have a time sequence attribute.

[0118] S302: Create an index for each content block according to a time sequence attribute of each content block in the first merged file, where the index includes a time index.

[0119] The time index may be understood as an index that includes time information in the index and is used to quickly locate a content block according to time.

[0120] Optionally, the time attributes (such as the start time and the end time) of each content block are analyzed, and an index entry is created for each content block, including a time index of the content block, wherein the time index includes the time attributes of the content block. These index entries are organized into an ordered structure, such as a time sequence, so that a specific content block can be quickly located based on the time information.

[0121] Write the created index information to a new file, the index file. This file can be in a format that is easy to parse and search, such as JSON, XML, or a custom binary format. The index file should contain enough information to accurately locate and read any content block in the first merged file when needed.

[0122] As an optional implementation, the step of establishing the index of each content block in step S302 includes steps B1-B2:

[0123] B1. Establish an offset index of each content block according to the offset of each content block in the first merged file.

[0124] The offset can be understood as the byte distance of a specific position (such as the starting position, etc.) of the content block in the first merged file relative to the starting position of the first merged file. If the offset of a content block is determined, the content block can be quickly located in the first merged file.

[0125] The offset index can be understood as a data structure that records the offset of each content block in the first merged file. Through this index, the physical position of a specific content block in the file can be quickly found without traversing the entire file.

[0126] Optionally, step B2 includes: first, based on a preset division rule, determining the boundary of each content block in the first merged file; second, for each content block, calculating the byte offset of its specific position relative to the starting position of the first merged file; then associating the offset of each content block with its corresponding identifier (such as the number or name of the content block) to form an index entry. All such index entries are combined to form an offset index. By establishing an offset index, the starting position of any content block in the file can be quickly located without traversing the entire file from the beginning.

[0127] B2. Establish a time index of the offset index according to the time sequence attribute of the first merged file.

[0128] As can be seen from the above content, each content block has its own specific timing attributes, such as start time and end time.

[0129] On the basis of the offset index, time information is further added to each index entry. Specifically, the timing attributes (start time and end time) of each content block are associated with its offset index. The time index points to the offset index, and the offset index points to the content block. These index entries with time information are organized in chronological order to form an orderly structure. In this way, a specific content block can be quickly located based on the time information.

[0130] By establishing a time index, the efficiency of content retrieval can be further improved, especially when it is necessary to quickly locate file content in chronological order.

[0131] S303: Generate an index file of the first merged file according to the index of each content block.

[0132] The index information of all content blocks is used according to the time sequence attributes of the content blocks to generate an index file of the first merged file. This index file can be used for subsequent rapid retrieval and access.

[0133] In the index file of the first merged file, the time index points to the offset index, and the offset index points to the offset of the location of the content block in the first merged file, and finally traversal and search are performed on the merged first merged file.

[0134] In this implementation, the first task not only generates the first merged file, but also creates a matching index file for it. This index file will greatly improve the efficiency in subsequent file access and operations, especially when it is necessary to quickly locate the file content in chronological order.

[0135] In this implementation, the task execution result also includes an index file of the first merged file.

[0136] S304: Merge the index files of the first merged files to generate an index file of the second merged file.

[0137] After the multiple first merged files are further merged into a second merged file, in order to maintain the consistency and integrity of the index, the index files of the first merged files also need to be merged.

[0138] Optionally, collect all index files of the first merged file. Then, sort and merge all index entries in chronological order according to the time indexes in these index files. During the merging process, it is necessary to handle the possible overlap or conflict of index entries. Finally, write the merged index information into a new index file, i.e., the index file of the second merged file, which contains the index information of the content blocks of all the first merged files and is organized in chronological order. This index file can be used for subsequent quick retrieval and access to the content blocks in the second merged file.

[0139] Through the above steps, an efficient file merging and indexing mechanism can be implemented, which not only generates the merged file, but also creates a matching index file for it, greatly improving the efficiency of subsequent file access and operations.

[0140] Figure 4 The following is a flow chart of the third file merging method provided in the embodiment of the present application. As an optional implementation method, Figure 4 As shown, the file merging method provided in this embodiment also includes steps S401-S403:

[0141] S401. In response to a query request, divide the query task corresponding to the query request into at least one query subtask according to a preset task division strategy, wherein the query request includes a query task for querying target data in a first file group, the first file group includes the second merged file, and the query subtask includes querying the target data in a second merged file.

[0142] The query request may be understood as a request issued by a user or a system to perform a specific data query on a certain data set (in this embodiment, the first file group, especially the second merged file therein).

[0143] The task division strategy can be understood as a preset rule or algorithm for decomposing a large query task into multiple small query subtasks that can be processed in parallel.

[0144] The query subtasks may be understood as small tasks split from the original query task, and each query subtask performs a query on a subset (eg, a second merged file) in the first file group.

[0145] Optionally, if the first file group consists of multiple second merged files, the query task may be split into query subtasks for each second merged file.

[0146] Optionally, if the data in the second merged file can be divided according to some logic (such as time range, ID range, etc.), the query task can be split into query subtasks for these data ranges.

[0147] S402: Execute the at least one query subtask in parallel to obtain a query result corresponding to each query subtask.

[0148] Parallel execution refers to the execution of multiple tasks or processes at the same time to increase overall processing speed.

[0149] The query result can be understood as the data or information obtained after each query subtask is executed, which is a part of the answer to the query request.

[0150] Optionally, use multi-threading or multi-processing technology to execute query subtasks simultaneously on multiple CPU cores.

[0151] S403: Merge the query results corresponding to each subtask using a preset merging rule to generate a final query result.

[0152] The merging rule can be understood as a preset logic or algorithm used to combine the results of multiple query subtasks into a unified and complete query result.

[0153] The final query result can be understood as the data or information that meets the user's original query request, obtained by merging the results of all query subtasks.

[0154] Optionally, if the results of the query subtasks are independent, they can be directly concatenated or merged into a large result set.

[0155] Optionally, if the results of the query subtasks may overlap, duplicate data needs to be removed when merging.

[0156] Optionally, if the final query results need to be presented in a certain order (such as time order, numerical value, etc.), the data needs to be sorted when merging.

[0157] Optionally, when the data volume of the query result is too large, the query result may be output in blocks to reduce the data output pressure caused by the excessive data volume.

[0158] Next, the query task in the embodiment of the present application is exemplarily described by taking the scenario of vehicle trajectory file merging and querying.

[0159] In a vehicle trajectory file merging and querying scenario, the file to be merged is a small vehicle trajectory file that matches the reporting frequency of the trajectory collection device (for example, a small vehicle trajectory file is generated every second). The file merging method provided by the embodiment of the present application is executed once every hour to merge all the small vehicle trajectory files within this hour to obtain a second merged file corresponding to the hour, that is, the second merged file is an hourly vehicle trajectory file.

[0160] In this scenario, for hourly trajectory data query, the query task is to query the vehicle trajectory data of a certain time period within a certain hour. It is only necessary to query the target data in the second merged file of the hour. Therefore, only one query subtask needs to be established for query, and the query result of the query subtask is the final query result. Exemplarily, the hourly trajectory data query includes the following steps: read the time index file of a specific hour, query the position of the minute-level shard in the time index file, and obtain the offset of the data stored in the trajectory file; read the corresponding trajectory data according to the obtained trajectory data file offset; integrate the read trajectory data and output it. For cross-hourly trajectory data query, the query task includes querying the vehicle trajectory data of part or all time periods within multiple hours. It is necessary to establish multiple query subtasks according to the number of hours spanned. Each query subtask corresponds to a data query of the second merged file (trajectory data within an hour), and the query results obtained after the execution of each query subtask are merged in chronological order to obtain the final query result. Exemplarily, querying trajectory data across the hourly level includes the following steps: parsing the start time and end time of the queried trajectory data, calculating the total number of natural hours from the start time to the end time, and providing a data basis for subsequent thread creation; generating a corresponding number of sub-threads according to the number of hours of the start and end time spans in the first step, for reading the trajectory data of each hour; each sub-thread first reads the trajectory index of each hour, and then filters out the offset of the trajectory data where the trajectory data to be read is located from the index; the sub-thread reads the trajectory data according to the data offset recorded in the index file; when each sub-thread finishes reading the trajectory data, the main thread merges the trajectory data of each sub-thread in order; when the main thread completes the data merging of the sub-threads, the trajectory data is output.

[0161] For the implementation of daily or multi-day trajectory data query, you can refer to the specific content of multi-hour trajectory data query.

[0162] Exemplary Devices

[0163] Corresponding to the above-mentioned file merging method, the embodiment of the present application also provides a file merging device. Figure 5 is a schematic diagram of the structure of a file merging device provided in an embodiment of the present application, such as Figure 5 As shown, the file merging device provided in the embodiment of the present application includes:

[0164] The first unit 501 is used to divide a plurality of to-be-merged files into a plurality of to-be-merged file groups according to a preset grouping strategy, wherein the grouping strategy includes a strategy for grouping based on the time sequence of the to-be-merged files;

[0165] The second unit 502 is used to call multiple sub-threads to respectively execute the first task to obtain multiple task execution results, wherein the first task includes merging the files to be merged in a group of files to be merged according to the time sequence of the files to be merged to generate a first merged file, and the task execution result includes the first merged file and the time sequence attribute of the first merged file, and each sub-thread executes at least one first task;

[0166] The third unit 503 is used to merge multiple first merged files according to the time sequence attribute of the first merged files to generate a second merged file.

[0167] The file merging device proposed in the present application divides multiple files to be merged into multiple file groups to be merged through a preset grouping strategy, and the files in each group are organized according to certain timing attributes. Subsequently, the file groups to be merged are assigned to different sub-threads, and each sub-thread is responsible for merging all files in at least one file group to generate a first merged file with clear timing attributes. Finally, according to the timing attributes of the first merged file, the first merged file is finally merged to generate a second merged file. On the one hand, the present application can maintain the time sequence of data during the merging process, ensure the consistency and accuracy of the merged files in terms of timing, and facilitate the execution of downstream tasks; on the other hand, it can significantly shorten the time required for file merging and improve the overall processing efficiency.

[0168] Optionally, the first unit 501 may be specifically used for:

[0169] Determine the number of target groups based on currently available processing resource information;

[0170] According to the time sequence attribute of each file to be merged, and in accordance with the time balance distribution principle and the same group distribution principle of adjacent time ranges, the multiple files to be merged are divided into the target grouping number of file groups to be merged.

[0171] Optionally, the first task also includes:

[0172] According to a preset division rule, the first merged file is logically divided into a plurality of content blocks;

[0173] According to the time sequence attribute of each content block in the first merged file, an index of each content block is established, wherein the index includes a time index;

[0174] Generate an index file of the first merged file according to the index of each content block;

[0175] The task execution result also includes an index file of the first merged file;

[0176] The third unit 503 can also be used for:

[0177] The index files of the first merged files are merged to generate an index file of the second merged file.

[0178] Optionally, the step of establishing an index for each content block includes:

[0179] Establishing an offset index of each content block according to the offset of each content block in the first merged file;

[0180] A time index of the offset index is established according to a time sequence attribute of the first merged file.

[0181] Optionally, the device further comprises:

[0182] The fourth unit is used to respond to a query request and divide the query task corresponding to the query request into at least one query subtask according to a preset task division strategy, wherein the query request includes a query task for querying target data in a first file group, the first file group includes the second merged file, and the query subtask includes querying the target data in a second merged file; execute the at least one query subtask in parallel to obtain a query result corresponding to each query subtask; and merge the query results corresponding to each subtask using a preset merging rule to generate a final query result.

[0183] The file merging device provided in this embodiment belongs to the same application concept as the file merging method provided in the above embodiments of this application, and can execute the file merging method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the file merging method. For technical details not fully described in this embodiment, please refer to the specific processing content of the file merging method provided in the above embodiments of this application, and will not be repeated here.

[0184] The functions implemented by the above first unit 501, second unit 502 and third unit 503 may be implemented by the same or different processors, respectively, and are not limited in the embodiment of the present application.

[0185] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0186] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0187] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0188] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0189] Exemplary Electronic Devices

[0190] Another embodiment of the present application also provides a file merging device, see Figure 6 As shown, the device includes:

[0191] Memory 200 and processor 210;

[0192] The memory 200 is connected to the processor 210 and is used to store programs;

[0193] The processor 210 is used to implement the file merging method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0194] Specifically, the file merging device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0195] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0196] A bus may include a pathway that transfers information between components of a computer system.

[0197] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0198] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0199] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0200] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0201] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0202] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0203] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any file merging method provided in the above embodiments of the present application.

[0204] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the file merging method introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the introduction to the above-mentioned embodiment of the file merging method.

[0205] Exemplary computer program products and storage media

[0206] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the file merging method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0207] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0208] In addition, the embodiment of the present application may also be a storage medium on which a computer program is stored. The computer program is executed by a processor to execute the steps of the file merging method according to various embodiments of the present application described in any of the above embodiments of this specification, and specifically the following steps may be implemented:

[0209] S201. Divide a plurality of to-be-merged files into a plurality of to-be-merged file groups according to a preset grouping strategy, wherein the grouping strategy includes a strategy for grouping the to-be-merged files based on a time sequence.

[0210] S202, calling multiple sub-threads to respectively execute the first task to obtain multiple task execution results, wherein the first task includes merging the files to be merged in a group of files to be merged according to the sequence of the files to be merged to generate a first merged file, and the task execution result includes the first merged file and the sequence attribute of the first merged file, and each sub-thread executes at least one first task.

[0211] S203: Merge multiple first merged files according to the time sequence attributes of the first merged files to generate a second merged file.

[0212] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0213] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0214] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0215] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0216] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0217] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0218] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0219] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0220] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0221] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0222] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A file merging method, characterized in that: include: According to a preset grouping strategy, the plurality of files to be merged are divided into a plurality of file groups to be merged, wherein the grouping strategy includes a strategy for grouping based on the time sequence of the files to be merged; Calling multiple sub-threads to respectively execute a first task to obtain multiple task execution results, wherein the first task includes merging files to be merged in a group of files to be merged according to the time sequence of the files to be merged to generate a first merged file, and the task execution result includes the first merged file and the time sequence attribute of the first merged file, and each sub-thread executes at least one first task; According to the time sequence attribute of the first merged file, multiple first merged files are merged to generate a second merged file.

2. The method according to claim 1, characterized in that The step of dividing the plurality of files to be merged into a plurality of file groups to be merged according to a preset grouping strategy includes: Determine the number of target groups based on currently available processing resource information; According to the time sequence attribute of each file to be merged, and in accordance with the time balance distribution principle and the same group distribution principle of adjacent time ranges, the multiple files to be merged are divided into the target grouping number of file groups to be merged.

3. The method according to claim 1, characterized in that The first task also includes: According to a preset division rule, the first merged file is logically divided into a plurality of content blocks; According to the time sequence attribute of each content block in the first merged file, an index of each content block is established, wherein the index includes a time index; Generate an index file of the first merged file according to the index of each content block; The task execution result also includes an index file of the first merged file; The method further comprises: The index files of the first merged files are merged to generate an index file of the second merged file.

4. The method according to claim 3, characterized in that The step of establishing the index of each content block includes: Establishing an offset index of each content block according to the offset of each content block in the first merged file; A time index of the offset index is established according to a time sequence attribute of the first merged file.

5. The method according to claim 1, characterized in that The method further comprises: In response to a query request, dividing the query task corresponding to the query request into at least one query subtask according to a preset task division strategy, wherein the query request includes a query task for querying target data in a first file group, the first file group includes the second merged file, and the query subtask includes querying the target data in a second merged file; Execute the at least one query subtask in parallel to obtain a query result corresponding to each query subtask; The query results corresponding to each subtask are merged using the preset merging rules to generate the final query result.

6. A file merging device, characterized in that: include: The first unit is used to divide the plurality of to-be-merged files into a plurality of to-be-merged file groups according to a preset grouping strategy, wherein the grouping strategy includes a strategy for grouping based on the time sequence of the to-be-merged files; The second unit is used to call multiple sub-threads to respectively execute the first task to obtain multiple task execution results, wherein the first task includes merging the files to be merged in a group of files to be merged according to the time sequence of the files to be merged to generate a first merged file, and the task execution result includes the first merged file and the time sequence attribute of the first merged file, and each sub-thread executes at least one first task; The third unit is used to merge multiple first merged files according to the time sequence attribute of the first merged files to generate a second merged file.

7. A file merging device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the file merging method as described in any one of claims 1 to 5 by running the program in the memory.

8. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the file merging method according to any one of claims 1 to 5 is implemented.

9. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to implement the file merging method as claimed in any one of claims 1 to 5.

10. A file merging system, characterized in that: include: The file merging device and the data storage module according to claim 7, wherein the data storage module is communicatively connected to the file merging device; The data storage module is used to store the second merged file generated by the file merging device.

Images