Method, device and equipment for writing lake surface data into cache and storage medium

Abstract

This specification relates to the technical field of lake surface data write caching, and provides a method, apparatus, device, and storage medium for lake surface data write caching. The method includes: converting a lake surface data set into a set of files in a target format as a set of files to be written; obtaining a configuration file for writing the set of files to be written to the cache; performing group processing on the set of files to be written according to the configuration file to determine the group to which each file to be written belongs and a corresponding JSON array for each group; invoking a write thread in a write thread pool, and assembling the data of the files to be written under the corresponding group into key-value pairs and writing them to the cache according to the JSON array. The embodiments of this specification can reduce the cost and improve the efficiency of lake surface data write caching.

Description

Technical Field

[0001] This specification relates to the technical field of lake surface data write caching, and in particular to a method, apparatus, device and storage medium for lake surface data write caching. Background Technology

[0002] A data lake is generally a repository or system that stores data in its raw format. In other words, unlike relational databases, data lakes typically store data as is (e.g., semi-structured data such as CSV, logs, XML, JSON) and unstructured data such as emails, documents, PDFs) without prior structuring. In a data lake, data is stored in the form of source tables (hereinafter referred to as lake tables). How to write lake table data to a cache in a low-cost and efficient manner is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] The purpose of this specification is to provide a method, apparatus, device, and storage medium for writing lake surface data to a cache, so as to reduce the cost of writing lake surface data to a cache and improve the efficiency of writing lake surface data to a cache.

[0004] To achieve the above objectives, in one aspect, embodiments of this specification provide a method for writing lake surface data to a cache, comprising:

[0005] Convert the lake surface data set into a set of files in the target format to serve as the set of files to be written;

[0006] Obtain the configuration file used to write the set of files to be written to the cache;

[0007] The set of files to be written is grouped according to the configuration file to determine the group to which each file belongs and the corresponding JSON array for each group;

[0008] The write thread in the write thread pool is invoked, and the data of the files to be written under the corresponding group is assembled into key-value pairs and written to the cache according to the json array.

[0009] In the method for writing lake surface data to the cache according to the embodiments of this specification, the step of grouping the set of files to be written according to the configuration file includes:

[0010] Based on the file storage path in the configuration file, scan the list of files to be written under that file storage path to generate a list of file names to be written.

[0011] The group to which each filename to be written belongs is determined based on the grouping information in the configuration file;

[0012] According to the configuration file, the configuration information of each file to be written in the same group is written into an inner JSON array, and the identifiers of the inner JSON arrays corresponding to all groups are written into an outer JSON array; wherein, each element in the inner JSON array is the configuration information of a single file to be written, and the elements in the inner JSON array are sorted according to the file execution order under the corresponding group.

[0013] In the method for writing lake surface data to the cache according to the embodiments of this specification, the step of calling the write thread in the write thread pool and assembling the data of the files to be written under the corresponding group into key-value pairs for writing to the cache according to the JSON array includes:

[0014] When the execution time for each group arrives, determine whether the group contains only one file to be written;

[0015] When the group contains multiple files to be written, the required number of threads for each file to be written under the group is obtained from the JSON array corresponding to the group, and the maximum required number of threads is identified.

[0016] Determine whether the number of available threads in the thread pool is not less than the maximum required number of threads;

[0017] When the number of available threads in the write thread pool is not less than the maximum required number of threads, the available threads are invoked to assemble the data of each file to be written under the group into key-value pairs and write them to the cache according to the configuration information in the JSON array corresponding to the group and the file execution order.

[0018] In the method for writing lake surface data to the cache according to the embodiments of this specification, the step of calling the write thread in the write thread pool and assembling the data of the files to be written under the corresponding group into key-value pairs for writing to the cache according to the JSON array further includes:

[0019] When the group contains only one file to be written, the required number of threads for that file to be written is obtained from the JSON array corresponding to the group;

[0020] Determine whether the number of available threads in the thread pool is not less than the required number of threads;

[0021] When the number of available threads in the write thread pool is not less than the required number of threads, the available threads are invoked to assemble the data of the file to be written into key-value pairs and write them to the cache according to the configuration information in the JSON array corresponding to the group.

[0022] In the method for writing lake surface data to the cache according to the embodiments of this specification, the configuration file includes cache parameters, general file parameters, and file execution parameters.

[0023] In the method for writing lake surface data to a cache in the embodiments of this specification, the cache includes a Redis cache cluster.

[0024] On the other hand, embodiments of this specification also provide an apparatus for writing lake surface data to a cache, comprising:

[0025] The conversion module is used to convert the lake table data set into a set of files in a target format, which can then be used as a set of files to be written.

[0026] The acquisition module is used to acquire the configuration file for writing the set of files to be written to the cache;

[0027] The grouping module is used to group the set of files to be written according to the configuration file, so as to determine the group to which each file to be written belongs and the JSON array corresponding to each group;

[0028] The calling module is used to invoke the write thread in the write thread pool and assemble the data of the files to be written under the corresponding group into key-value pairs and write them to the cache based on the JSON array.

[0029] On the other hand, embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the computer program, when run by the processor, executes instructions for the above-described method.

[0030] On the other hand, embodiments of this specification also provide a computer storage medium storing a computer program thereon, which, when run by the processor of a computer device, executes instructions for the above-described method.

[0031] On the other hand, embodiments of this specification also provide a computer program product, which includes a computer program that, when run by the processor of a computer device, executes instructions for the above-described method.

[0032] As can be seen from the technical solutions provided in the embodiments of this specification above, in these embodiments, since the set of files to be written can be grouped according to the configuration file to determine the group to which each file to be written belongs and the corresponding JSON array for each group, and the write thread in the write thread pool can be called, and the data of the files to be written under the corresponding group can be assembled into key-value pairs and written to the cache according to the JSON array, it is no longer necessary to write program code and test separately for writing caches for different lake table files. Therefore, the cost of writing caches to lake table data can be greatly reduced and the efficiency of writing caches to lake table data can be improved. In addition, the efficiency of writing caches to lake table data is further improved by utilizing thread pool technology. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0034] Figure 1 This specification illustrates application scenarios of writing lake surface data to the cache in some embodiments.

[0035] Figure 2 This specification shows flowcharts illustrating methods for writing lake surface data to a cache in some embodiments;

[0036] Figure 3 It shows Figure 2 The flowchart shown illustrates the process of grouping files to be written according to the configuration file.

[0037] Figure 4 It shows Figure 2 The flowchart shown illustrates how the data from the corresponding group of files to be written is assembled into key-value pairs and written to the cache based on the JSON array.

[0038] Figure 5 This specification illustrates a schematic diagram of the process of writing lake table data to the cache in an exemplary embodiment.

[0039] Figure 6 This specification shows a structural block diagram of an apparatus for writing lake surface data to a cache in some embodiments;

[0040] Figure 7 Block diagrams of computer devices in some embodiments of this specification are shown.

[0041] [Explanation of Labels in the Attached Image]

[0042] 10. Data Lake;

[0043] 20. Server;

[0044] 30. Caching;

[0045] 61. Conversion module;

[0046] 62. Acquisition module;

[0047] 63. Grouped modules;

[0048] 64. Call the module;

[0049] 702. Computer equipment;

[0050] 704, Processor;

[0051] 706. Memory;

[0052] 708. Drive mechanism;

[0053] 710. Input / output interfaces;

[0054] 712. Input devices;

[0055] 714. Output devices;

[0056] 716. Presentation equipment;

[0057] 718. Graphical User Interface;

[0058] 720. Network interface;

[0059] 722. Communication link;

[0060] 724. Communication bus. Detailed Implementation

[0061] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0062] This specification describes a technique for writing lake table data into a cache, which can be applied to any big data application scenario, including the fintech field, and is not limited thereto. In some embodiments, taking the fintech field as an example, there is often a large amount of data with low processing timeliness requirements, which is not stored in a relational database but only in a lake table. Sometimes, due to business logic requirements, the program needs to query this data in milliseconds, which requires loading certain data from the lake table into the cache in a certain way. In this case, since it often involves the processing of data in different original formats, separate program code for writing to the cache needs to be developed and tested for different lake table files, resulting in high cost and low efficiency of writing lake table data into the cache. In view of this, this specification provides a new technical solution for writing lake table data into the cache to reduce the cost and improve the efficiency of writing lake table data into the cache.

[0063] Figure 1The following describes an application scenario of writing lake surface data to a cache in some embodiments of this specification. In this application scenario, server 20 can convert the lake surface data set in data lake 10 into a set of files in a target format and write it to a specified directory as a set of files to be written; obtain a configuration file for writing the set of files to be written to the cache; perform group processing on the set of files to be written according to the configuration file to determine the group to which each file to be written belongs and the JSON array corresponding to each group; call the write thread in the write thread pool, and assemble the data of the files to be written under the corresponding group into key-value pairs according to the JSON array and write them to cache 30 (e.g., distributed cache).

[0064] This specification provides an embodiment of a method for writing lake surface data to a cache, which can be applied to the aforementioned server side. (Refer to...) Figure 2 As shown, in some embodiments, the method of writing lake surface data to the cache may include the following steps:

[0065] Step 201: Convert the lake surface data set into a set of files in the target format to serve as the set of files to be written.

[0066] Step 202: Obtain the configuration file used to write the set of files to be written to the cache.

[0067] Step 203: Group the set of files to be written according to the configuration file to determine the group to which each file to be written belongs and the corresponding JSON array for each group.

[0068] Step 204: Call the write thread in the write thread pool, and assemble the data of the files to be written under the corresponding group into key-value pairs and write them to the cache according to the json array.

[0069] In the embodiments of this specification, since the set of files to be written can be grouped according to the configuration file to determine the group to which each file belongs and the corresponding JSON array of each group, and the write thread in the write thread pool can be called to assemble the data of the files to be written under the corresponding group into key-value pairs for writing to the cache according to the JSON array, it is no longer necessary to write program code and test separately for writing caches of different lake table files. Therefore, the cost of writing caches of lake table data can be greatly reduced and the efficiency of writing caches of lake table data can be improved. In addition, the efficiency of writing caches of lake table data is further improved by utilizing thread pool technology.

[0070] A lake table dataset is a collection of one or more lake tables retrieved from a data lake on demand (e.g., for business processing needs). Combined with... Figure 5As shown, to facilitate faster processing, the lake table files in the lake table dataset can be uniformly converted into target format files (the specific format can be specified by the user), thus forming a set of target format files as the set of files to be written. For example, in an exemplary embodiment, the lake table files in the lake table dataset can be uniformly converted into Hadoop Distributed File System (HDFS) files.

[0071] The configuration file can be configured by the user through the client and provided to the server; the configuration file contains configuration information configured to implement writing the set of files to be written to the cache.

[0072] In some embodiments, the configuration file mainly includes cache parameters, general file parameters, and file execution parameters. Cache parameters may include, but are not limited to, specifying which caches the file to be written should be placed in (multiple caches can be written to); general file parameters may include, but are not limited to, specifying the file storage path of the file to be written (i.e., specifying the file directory where the program reads the file to be written); file execution parameters may include, but are not limited to, how each file content is assembled into the cache command, how placeholders are used, the file cleanup cycle, and how many execution threads are required. For example, in an exemplary embodiment, the configuration information in the configuration file can be shown in Table 1 below (here, the cache is a Redis cache cluster as an example).

[0073] Table 1

[0074]

[0075]

[0076]

[0077] The set of files to be written typically contains multiple files. Therefore, it's necessary to group the set of files according to the configuration file to determine the group to which each file belongs and the corresponding JSON array for each group. Thus, grouping not only involves grouping the files within the set but also integrating the configuration information for each file within each group. A JSON array is a dynamic array that describes configuration information in JSON format. Using JSON arrays not only makes it easier for users to write, read, and understand configuration information but also facilitates program processing.

[0078] refer to Figure 3 As shown, in some embodiments, grouping the set of files to be written according to the configuration file may include the following steps:

[0079] Step 301: Based on the file storage path in the configuration file, scan the list of files to be written under that file storage path to form a list of file names to be written.

[0080] Combination Figure 5 As shown, the specified directory storing the set of files to be written can be found based on the file storage path. Then, all list items in the list of files to be written under that file storage path are scanned to obtain the file prefix name information (fileParam.fileName). Based on the obtained file prefix name (here referring to the file name), a list of file names to be written is formed. For example, in an exemplary embodiment, the list of files to be written contains four files: File1.BIN, File2.BIN, File3.BIN, and File4.BIN; then File1, File2, File3, and File4 can form a list of file names to be written.

[0081] Step 302: Determine the group to which each filename to be written belongs in the list of filenames to be written based on the grouping information in the configuration file.

[0082] The group information in the configuration file specifies the group to which the files to be written belong; therefore, the group to which each file in the list of filenames to be written belongs can be identified based on the group information in the configuration file. When there are multiple files to be written within the same group, there is a defined file execution order (i.e., execution priority) among these multiple files to be written.

[0083] Step 303: Based on the configuration file, write the configuration information of each file to be written within the same group into an inner JSON array, and write the identifiers of the inner JSON arrays corresponding to all groups into an outer JSON array. Each element in the inner JSON array represents the configuration information of a single file to be written, and the elements in the inner JSON array are ordered according to the file execution order under the corresponding group. By nesting the outer and inner JSON arrays, data relationships can be expressed more effectively, thereby improving code readability and maintainability; and making it easier to handle complex or storage-intensive application scenarios.

[0084] For example, in an exemplary embodiment, the set of files to be written is {File1, File2, File3, and File4}, and the corresponding grouping is group1: {File1, File2}, group2: {File4, File3}; where the identifier of the inner JSON array corresponding to group1 is Array1, and the identifier of the inner JSON array corresponding to group2 is Array2; then:

[0085] The outer JSON array can be represented as:

[0086] [Array1,Array2]

[0087] The inner JSON array Array1 can be represented as:

[0088]

[0089] The inner JSON array Array2 can be represented as:

[0090]

[0091] The inner JSON array contains elements representing specific configuration information (configuration parameters), and the configuration information on the same line corresponds to the configuration information in the file to be written; for example, p 11 ,p 12 ,L,p 1n For the configuration information of File1, p 21 ,p 22 ,L,p 2n This is the configuration information for File1.

[0092] In some embodiments, to improve write efficiency, thread pool technology can be used to enable parallel execution among different groups. (See reference) Figure 4 As shown, in some embodiments, invoking a write thread in the write thread pool and assembling the data of the files to be written under the corresponding group into key-value pairs for writing to the cache according to the JSON array may include the following steps:

[0093] Step 401: Monitor when the group execution time arrives.

[0094] In some embodiments, each group can be treated as an independent timed task, and by monitoring the timed interval, it can be determined whether the execution time has arrived.

[0095] Step 402: Determine if the group contains only one file to be written; if not, proceed to step 403; if yes, proceed to step 406.

[0096] For each group, when its execution time arrives, it can be determined whether the group contains only one file to be written.

[0097] Step 403: Obtain the required number of threads for each file to be written under the group from the JSON array corresponding to the group, and identify the maximum required number of threads.

[0098] In the embodiments of this specification, when there are multiple files to be written under the same group, these files need to be executed serially according to the configured file execution order. To ensure transaction consistency, the writing of all files to be written under the same group can be treated as a single transaction. That is, the group is considered to have succeeded only when all files to be written under the same group have been successfully written; if any file to be written under the same group fails to be written, the group is considered to have failed. Since each file to be written under the same group is configured with a separate number of demand threads, the maximum number of demand threads should be used as the basis for comparison to ensure transaction consistency.

[0099] Step 404: Determine whether the number of available threads in the write thread pool is not less than the maximum required number of threads. If the number of available threads in the write thread pool is not less than the maximum required number of threads, proceed to step 405; otherwise, proceed to step 401 (i.e., wait for the next execution time to arrive and try again).

[0100] The total number of threads in the write thread pool can be determined by referencing the cache performance settings to ensure stable cache operation. Furthermore, when determining the number of available threads in the write thread pool, a distributed lock can be used to ensure that the number of used threads in the write thread pool is not modified when calculating available threads.

[0101] Step 405: Invoke an available thread to assemble the data of each file to be written under the group into a key-value pair and write it to the cache according to the configuration information in the JSON array corresponding to the group and the file execution order.

[0102] For each thread, it can assemble keys according to some configuration information in the JSON array and assemble values ​​according to other configuration information in the JSON array, thus generating key-value pairs.

[0103] In some embodiments, key-value pairs can be assembled in a manner similar to "prefix + placeholder + suffix". For example, in an exemplary embodiment, a thread can take "keyChope":"$","keyRule":"keySample$","keyReplace":[0] from the json array to assemble the key, resulting in keySamplevalue1; and take "valueChope":"$","valueRule":"value$","valueReplace" from the json array to assemble the value, resulting in valuevalue2.

[0104] Step 406: Obtain the required number of threads for the file to be written from the JSON array corresponding to the group.

[0105] Step 407: Determine whether the number of available threads in the write thread pool is not less than the required number of threads. If the number of available threads in the write thread pool is not less than the required number of threads, proceed to step 408; otherwise, proceed to step 401.

[0106] Step 408: Invoke an available thread to assemble the data to be written to the file into key-value pairs and write them to the cache according to the configuration information in the JSON array corresponding to the group.

[0107] It should be noted that for each group, after each time a thread completes writing to the file buffer, the thread should be released in a timely manner so that the number of available threads in the write thread pool can be updated promptly.

[0108] Although the process described above includes multiple operations that occur in a specific order, it should be clearly understood that these processes may include more or fewer operations, which may be executed sequentially or in parallel (e.g., using parallel processors or a multithreaded environment).

[0109] Corresponding to the above-described method for writing lake surface data to a cache, this specification also provides an apparatus for writing lake surface data to a cache, which can be configured on the aforementioned server. (See reference...) Figure 6 As shown, in some embodiments, the means for writing lake surface data to the cache may include:

[0110] The conversion module 61 is used to convert the lake surface data set into a set of files in a target format, so as to serve as a set of files to be written.

[0111] The acquisition module 62 is used to acquire a configuration file for writing the set of files to be written to the cache;

[0112] Grouping module 63 is used to group the set of files to be written according to the configuration file to determine the group to which each file to be written belongs and the JSON array corresponding to each group;

[0113] Module 64 is invoked to call the write thread in the write thread pool and assemble the data of the files to be written under the corresponding group into key-value pairs for writing to the cache based on the json array.

[0114] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0115] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this specification are all information and data authorized and agreed upon by the user and fully authorized by all parties.

[0116] Embodiments of this specification also provide a computer device. For example... Figure 7 As shown, in some embodiments of this specification, the computer device 702 may include one or more processors 704, such as one or more central processing units (CPUs) or graphics processing units (GPUs), each of which may implement one or more hardware threads. The computer device 702 may also include any memory 706 for storing information of any kind, such as code, settings, data, etc. In one specific embodiment, a computer program is stored on the memory 706 and can run on the processor 704. When the computer program is run by the processor 704, it can execute instructions for the lake surface data writing cache method described in any of the above embodiments. Without limitation, for example, the memory 706 may include any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 702. In one case, when the processor 704 executes associated instructions stored in any memory or combination of memories, the computer device 702 can perform any operation of the associated instructions. The computer device 702 also includes one or more drive mechanisms 708 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0117] Computer device 702 may also include an input / output interface 710 (I / O) for receiving various inputs (via input device 712) and providing various outputs (via output device 714). A specific output mechanism may include a presentation device 716 and an associated graphical user interface 718 (GUI). In other embodiments, the input / output interface 710 (I / O), input device 712, and output device 714 may be omitted, and the device may function solely as a computer device within a network. Computer device 702 may also include one or more network interfaces 720 for exchanging data with other devices via one or more communication links 722. One or more communication buses 724 couple the components described above together.

[0118] Communication link 722 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0119] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), computer-readable storage media, and computer program products according to some embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processor to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processor, create a mechanism for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0120] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processor to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0121] These computer program instructions may also be loaded onto a computer or other programmable data processor, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0122] In a typical configuration, a computer device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0123] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0124] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by computer equipment. As defined in this specification, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0125] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] The embodiments described in this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processors connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0127] It should also be understood that, in the embodiments of this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0128] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0129] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0130] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for writing lake surface data into a cache, characterized in that, include: Convert the lake surface data set into a set of files in the target format to serve as the set of files to be written; Obtain the configuration file used to write the set of files to be written to the cache; The set of files to be written is grouped according to the configuration file to determine the group to which each file belongs and the corresponding JSON array for each group; Call the write thread in the write thread pool, and assemble the data of the files to be written under the corresponding group into key-value pairs and write them to the cache according to the json array; The step of grouping the set of files to be written according to the configuration file includes: Based on the file storage path in the configuration file, scan the list of files to be written under that file storage path to generate a list of file names to be written. The group to which each filename to be written belongs is determined based on the grouping information in the configuration file; According to the configuration file, the configuration information of each file to be written in the same group is written into an inner JSON array, and the identifiers of the inner JSON arrays corresponding to all groups are written into an outer JSON array; wherein, each element in the inner JSON array is the configuration information of a single file to be written, and the configuration information in the same line corresponds to the configuration information of the same file to be written; the elements in the inner JSON array are sorted according to the file execution order under the corresponding group; The step of invoking the write thread in the write thread pool and assembling the data of the files to be written under the corresponding group into key-value pairs for writing to the cache according to the JSON array includes: When the execution time for each group arrives, determine whether the group contains only one file to be written; When the group contains multiple files to be written, the required number of threads for each file to be written under the group is obtained from the JSON array corresponding to the group, and the maximum required number of threads is identified; wherein, the multiple files to be written are executed serially according to the configured file execution order; Determine whether the number of available threads in the thread pool is not less than the maximum required number of threads; When the number of available threads in the write thread pool is not less than the maximum required number of threads, the available threads are invoked to assemble the data of each file to be written under the group into key-value pairs and write them to the cache according to the configuration information in the JSON array corresponding to the group and the file execution order.

2. The method for writing lake surface data to the cache as described in claim 1, characterized in that, The step of invoking the write thread in the write thread pool and assembling the data of the files to be written under the corresponding group into key-value pairs for writing to the cache according to the JSON array also includes: When the group contains only one file to be written, the required number of threads for that file to be written is obtained from the JSON array corresponding to the group; Determine whether the number of available threads in the thread pool is not less than the required number of threads; When the number of available threads in the write thread pool is not less than the required number of threads, the available threads are invoked to assemble the data of the file to be written into key-value pairs and write them to the cache according to the configuration information in the JSON array corresponding to the group.

3. The method for writing lake surface data to the cache as described in claim 1, characterized in that, The configuration file contains cache parameters, general file parameters, and file execution parameters.

4. The method for writing lake surface data to the cache as described in claim 1, characterized in that, The cache includes a Redis cache cluster.

5. An apparatus for writing lake surface data to a cache, characterized in that, include: The conversion module is used to convert the lake table data set into a set of files in a target format, which can then be used as a set of files to be written. The acquisition module is used to acquire the configuration file for writing the set of files to be written to the cache; The grouping module is used to group the set of files to be written according to the configuration file, so as to determine the group to which each file to be written belongs and the JSON array corresponding to each group; The calling module is used to call the write thread in the write thread pool and assemble the data of the files to be written under the corresponding group into key-value pairs and write them to the cache according to the json array; Specifically, the grouping module is used to scan the list of files to be written under the file storage path in the configuration file to generate a list of file names to be written. The group to which each filename to be written belongs is determined based on the grouping information in the configuration file; According to the configuration file, the configuration information of each file to be written in the same group is written into an inner JSON array, and the identifiers of the inner JSON arrays corresponding to all groups are written into an outer JSON array; wherein, each element in the inner JSON array is the configuration information of a single file to be written, and the configuration information in the same line corresponds to the configuration information of the same file to be written; the elements in the inner JSON array are sorted according to the file execution order under the corresponding group; Specifically, the calling module is used to determine whether a group contains only one file to be written when the execution time of each group arrives; when the group contains multiple files to be written, it obtains the required number of threads for each file to be written under the group from the JSON array corresponding to the group, and identifies the maximum required number of threads; wherein the multiple files to be written are executed serially according to the configured file execution order; it determines whether the number of available threads in the writing thread pool is not less than the maximum required number of threads; when the number of available threads in the writing thread pool is not less than the maximum required number of threads, it calls the available threads to assemble the data of each file to be written under the group into a key-value pair and write it to the cache according to the configuration information in the JSON array corresponding to the group and the file execution order.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the computer program is run by the processor, it executes the instructions of the method according to any one of claims 1-4.

7. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor of the computer device, it executes the instructions of the method according to any one of claims 1-4.

8. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, performs instructions according to any one of claims 1-4.

Images