Data Management Method, System, Device, Program Product, and Storage Medium

Abstract

Embodiments of the present invention disclose a data management method, system, device, program product, and medium. The data management method includes: receiving an export request and adding the export request to an export request queue; adding the export request in the export request queue to an export execution queue; if the export execution queue is not empty and currently does not contain an ongoing and unfinished export task, taking out the corresponding export request from the export execution queue, and establishing a corresponding export task according to the data export type supported by the data receiving object included in the export request; according to the export task, executing an export state machine, and sending the export data to the data receiving object through an associated interface corresponding to the data receiving object hooked to the data management system.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a data management method, system, device, computer program product, and computer-readable storage medium. Background Art

[0002] Currently, for data export of various known business systems, taking log data as an example, the main method is that a user issues a data export request by operating the business system, creates a corresponding export task according to the export request, calls the business system corresponding to the export task to obtain data from the business system server, and integrates the data into a file for storage to export.

[0003] In the process of implementing the present invention, the inventors found that the prior art has at least the following problems: Before exporting data, it is necessary to integrate the data into a file and store it in the form of a file for export. This data export method is relatively limited and is only limited to a specific type of data export, and cannot flexibly export according to different requirements of the business system, which is not conducive to the management of data export tasks. Summary of the Invention

[0004] To solve the existing technical problems, embodiments of the present invention provide a data management method, system, device, computer program product, and computer-readable storage medium with a more flexible data export method and conducive to effectively improving the data export management ability.

[0005] The technical solution of the embodiments of the present invention is implemented as follows:

[0006] In a first aspect, a data management method is provided, including:

[0007] Receiving an export request and adding the export request to an export request queue;

[0008] Adding the export request in the export request queue to an export execution queue;

[0009] If the export execution queue is not empty and does not currently contain an ongoing and unfinished export task, taking out the corresponding export request from the export execution queue, and establishing a corresponding export task according to the data export type supported by the data receiving object included in the export request;

[0010] According to the export task, executing an export state machine, and sending export data to the data receiving object through an associated interface of the corresponding data receiving object hooked to the data management system.

[0011] In a second aspect, a data management system is provided, including:

[0012] A request queue module, configured to receive an export request and add the export request to an export request queue;

[0013] An execution queue module, configured to add the export request in the export request queue to an export execution queue;

[0014] An export service module, configured to, if the export execution queue is not empty and currently does not include an ongoing and unfinished export task, take out the corresponding export request from the export execution queue, and establish a corresponding export task according to the data export type supported by the data receiving object included in the export request;

[0015] An export execution module, configured to execute an export state machine according to the export task, and send export data to the data receiving object through an associated interface of the corresponding data receiving object hooked to the data management system.

[0016] In a third aspect, a data management device is provided, including a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor is caused to execute the steps of the data management method according to any embodiment of the present application.

[0017] In a fourth aspect, a computer program product is provided. When the instructions in the computer program product are executed by a processor of a data management device, the data management device is caused to execute the steps of the data management method according to any embodiment of the present application.

[0018] In a fifth aspect, a computer-readable storage medium is provided, storing a computer program, characterized in that when the computer program is executed by a processor, the processor is caused to execute the steps of the data management method according to any embodiment of the present application.

[0019] The data management method provided by the above embodiments of the present application has at least the following beneficial effects:

[0020] Through the export request queue and the export execution queue, the loss of export requests in the case of multiple export requests existing simultaneously can be avoided, and multiple export tasks can be compactly completed according to the priorities of the export requests through the export execution state machine, improving the data export efficiency; multiple data receiving objects supporting different data export types can be included in the same request, and multiple data receiving objects supporting the same data export type can be included in the same export task. By hooking the associated interfaces of the corresponding data receiving objects to the data management system, the data management system sends data to the associated interfaces, providing only one service for exporting data, and completing different forms of export through external actual execution interfaces, rather than integrating the data into a file and storing and exporting it in the form of a file. Therefore, the data export method can be more flexible, effectively improving the ability to manage data export.

[0021] The data management system, device, computer program product, and computer-readable storage medium provided by the above embodiments respectively belong to the same concept as the corresponding data management method embodiments, and thus respectively have the same technical effects as the corresponding data management method embodiments, which will not be elaborated here. Description of the Drawings

[0022] Figure 1 It is a system architecture diagram of an optional application scenario of the data management method in an embodiment;

[0023] Figure 2 It is a flowchart of the data management method in an embodiment;

[0024] Figure 3 It is a flowchart of a file-type export task in an embodiment;

[0025] Figure 4 It is a flowchart of a communication-type export task in an embodiment;

[0026] Figure 5 It is a data format diagram of structured data extraction information in an embodiment;

[0027] Figure 6 It is a data export flowchart taking the exported data as a log record as an example in an embodiment;

[0028] Figure 7 It is a flowchart of the data management method in an optional example;

[0029] Figure 8 It is a schematic diagram of an export state machine in an embodiment;

[0030] Figure 9 It is a structure diagram of a data management system in an embodiment;

[0031] Figure 10 It is a structure diagram of a data management device in an embodiment. Detailed Embodiments

[0032] The technical solution of the present invention will be further elaborated in detail below in conjunction with the accompanying drawings of the specification and specific embodiments.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the protection scope of the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0034] In the following description, the expression "some embodiments" describes a subset of all possible embodiments. However, it should be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0035] Please refer to Figure 1 , which is an optional application scenario of the data management method provided by the embodiments of the present application. The data system may include a business system 11 and a data management system 12. Among them, the business system 11 refers to the main body that generates exported data, such as log records. The data management system 12 may refer to the main body that exports the data that meets the export request through the data management method in the embodiments of the present application according to the received user's export request for data. The business system 11 and the data management system 12 may refer to two logically independent software. The data management system 12 may set an interface for the business system to call, and actively obtain or start according to user operations to obtain the data management permission for the business system through system permissions. Optionally, the business system 11 and the data management system 12 may also be a software integrated logically as a whole. The computer program that implements the data management method in the embodiments of the present application when executed by a processor is integrated into the business system 11, so that the business system 11 itself can export the data that meets the export request through the data management method in the embodiments of the present application according to the received user's export request for data. It should be noted that the business system 11 may be replaced by other software systems such as system programs and applets. Optionally, the data system further includes a storage system 13. The storage system 13 refers to an object that cooperates with the data management system 12 to provide the data management system 12 to complete operations such as writing, querying, and reading the data part of the exported data, such as log records. For example, the storage system 13 may be a storage space composed of flash blocks or sectors as storage management units, or various internal or external memories that support read / write operations.

[0036] In an optional example, the business system 11 and the data management system 12 are respectively independent software systems. The data such as log records and work data generated during the operation of the business system 11 can call the interface of the data management system 12 to manage its log records, work data and other data through the data management system 12.

[0037] Please refer to Figure 2 , for the data management method provided by an embodiment of the present application, which can be applied to Figure 1 the data management system in, and the data management method includes the following steps:

[0038] S101, receive an export request and add the export request to an export request queue.

[0039] The export request can be used to represent that the user needs to perform data export and contains relevant information for determining the data to be exported. The data management system can provide a data export operation interface. The user can perform operations in the data export operation interface, and the receiving of the export request can be that the data management system receives the export request according to the user's operations in the data export operation interface. Optionally, the receiving of the export request can also be that the business system generates an export request according to the user's operations and sends the export request to the data management system. Adding the export request to the export request queue can complete the corresponding data export task by taking out the export request from the export request queue. The export request queue can be used to save all the currently newly received export requests that have not entered the execution state, which can avoid the loss of export requests when there are multiple export requests in the same time period. Optionally, the export request queue can be a circular queue. After the data management system receives an export request, the export request is stored in the export request queue according to the first-in, first-out rule.

[0040] S103. Add the export request in the export request queue to the export execution queue.

[0041] The export execution queue can be used to save the export requests that are about to enter the execution state. In an optional example, there is no more than one export request in the export execution queue in the same time period. The data management system continuously queries the export request queue through a thread. If the export request queue is not empty and the export execution queue is empty, the export request with the highest current priority is taken out from the export request queue and added to the export execution queue; if the export request queue is not empty and the export execution queue is not empty, taking out the export request from the export request queue and adding it to the export execution queue is postponed. In this embodiment, the export request with the highest current priority in the export request queue is the earliest added export request among the currently remaining export requests in the export request queue.

[0042] S105. If the export execution queue is not empty and does not currently contain an ongoing and unfinished export task, take out the corresponding export request from the export execution queue, and establish a corresponding export task according to the data export type supported by the data receiving object included in the export request.

[0043] The data export type refers to the form of exporting data that meets the current export request, such as exporting data to a file or sending data to the corresponding target device in a communication manner. The data receiving object refers to the object to which the exported data is written. For example, in the form of exporting data to a file, the data receiving object is the file system; in the form of sending data to the corresponding target device in a communication manner, the data receiving object is the target device. The export task corresponds to different data export types supported by the data receiving object included in the same export request. For example, for the data receiving object with the data export type of exporting data to a file in the same export request, a corresponding file-type export task can be established; for the data export type of sending data to the corresponding target device in a communication manner, a corresponding communication-type export task can be established.

[0044] The data management system can continuously query the export execution queue through a thread. If the export execution queue is not empty and the export state machine does not contain an ongoing and unfinished export task, the corresponding export request is taken out from the export execution queue, and a corresponding export task is established according to the data export type supported by the data receiving object included in the export request. If the export execution queue is empty and the export request queue is not empty, the export request with the highest current priority is taken out from the export request queue and added to the export execution queue.

[0045] S107, according to the export task, execute the export state machine, and send the exported data to the data receiving object through the associated interface of the corresponding data receiving object hooked to the data management system.

[0046] Before exporting data by attaching to the associated interface corresponding to the data receiving object of the data management system, it further includes attaching the associated interface corresponding to the data receiving object to the data management system. The data management system executes the export state machine according to the export task, and exports the data through the associated interface of the data receiving object, so that the data management system is only responsible for packing the format of the export data that meets the current export request, and executes the data export through the external actual execution interface, so that it is not limited to a specific type of data export. Among them, an interface refers to a class or function in a program that is responsible for transmitting or receiving data between different program modules and performing processing. Common interfaces include API interfaces (Application Programming Interface). The associated interface of the data receiving object refers to the relevant interfaces that need to be used in the process of receiving the export data and writing the export data into the target file. Attaching refers to a method of replacing the call of the original function with the call of the target function. For example, common attaching methods can include modifying the call code in the original function, modifying the address of the function in the import table of the PE (Portable Executable) file, etc.

[0047] In the above embodiments, in the process of the data management method implementing data export, through the export request queue and the export execution queue, the loss of export requests can be avoided in the case of multiple export requests existing at the same time, and through the export execution state machine, multiple export tasks can be compactly completed according to the priority of the export requests, improving the data export efficiency; multiple data receiving objects supporting different data export types can be included in the same request, and multiple data receiving objects supporting the same data export type can be included in the same export task. By attaching the associated interfaces corresponding to the data receiving objects to the data management system, the data management system only provides one export service and completes the export through the external actual execution interface, so that the data export method can be more flexible and effectively improve the ability to manage data export; the data management system is implemented through two queues and a state machine, forming a standard data structure, so that the carrier of the data management method has no strict requirements on the implementation platform, and the method has strong portability.

[0048] In some embodiments, S107, according to the export task, execute the export state machine, and send the export data to the data receiving object through the associated interface corresponding to the data receiving object attached to the data management system, including:

[0049] Attach the open interface, write interface, and close interface of the file system to the data management system;

[0050] If the export task is a file-based export task, according to the file-based export task, execute the export state machine, and call the open interface, write interface, and close interface to write the export data into the target file.

[0051] For a data receiving object in the same export request whose data export type is to export data to a file, establish a corresponding file-based export task. In the form of exporting data to a file, the data receiving object is the file system, and the associated interfaces of the data receiving object correspondingly include the open interface, write interface, and close interface of the file system. After the data management system receives the export request, it creates a corresponding export task. If the export state machine does not currently contain an ongoing and unfinished export task, the data management system automatically executes the entire export state machine, and calls the open interface, write interface, and close interface mounted by the file system to write the export data into the target file.

[0052] Please refer to Figure 3 , in an optional example, the export execution process of the file-based export task includes: S11, mount the file open interface of the file system to the data management system; S12, mount the file write interface of the file system to the data management system; S13, mount the file close interface of the file system to the data management system; S14, externally call the request interface of the data management system to submit an export request once; where the external party can refer to a business system that authorizes the data management system to manage its log records, work data, etc.; S15, create an export task according to the export request, execute the export state machine, and call the file open interface, file write interface, and file close interface of the file system during the process to write the export data that meets the export request into the target file. Optionally, the data management system is an export system that executes an independent data export function.

[0053] In the above embodiments, the data management system is used to provide data export services, not limited to a specific form of data export type. When the export request contains a data receiving object whose data export type is to export data to a file, by mounting the relevant execution interfaces of the file system, data export in file mode is realized to meet and not be limited to the data export requirements in file mode.

[0054] Optionally, S101, receiving an export request and adding the export request to the export request queue includes:

[0055] Obtain export configuration information through the request interface according to the export configuration operation, and submit an export request based on the export configuration information;

[0056] Add the export request to the export request queue.

[0057] Among them, the data management system can provide a data export operation interface, which provides multiple configuration items and a request submission button through the data export operation interface. Each configuration item can be selected by the user to set corresponding configuration information, and the request submission button can be selected by the user after completing the settings of each configuration item to confirm the submission of the export request. The data management system obtains the export configuration information according to the export configuration operation of the user in the data export operation interface through the request interface and submits the export request, and adds the export request to the export request queue. Each configuration item can be set according to actual needs. For example, it can include a configuration item for setting the data receiving object to set the file export method for exporting to the target file and the export method for sending data to one or more target devices in a communication manner; it can include a configuration item for setting the export data range to set which types of data need to be exported or which time period of data needs to be exported, etc. Each configuration item can be set as a drop-down option, and multiple options for the corresponding configuration information are provided through the drop-down option for the user to select from the options to complete the setting of the corresponding configuration information; or, each configuration item can also be set as a data input area, and the corresponding configuration information can be set by inputting in the data input area according to a preset format.

[0058] In the above embodiments, the data management system provides the configuration of the information of the export request, which is beneficial to simplifying the user operation, reducing the operation difficulty of the data management system, and facilitating meeting more different data export requirements according to the user's configuration operation.

[0059] In some embodiments, S107, the execution of the export state machine according to the export task and sending the export data to the data receiving object through the association interface corresponding to the data receiving object hooked to the data management system includes:

[0060] Hook the communication interface to the data management system;

[0061] If the export task is a communication-type export task, execute the export state machine according to the communication-type export task, call the communication interface to send the export data to the target device, and the target device receives the export data and writes it into the target file.

[0062] Among them, for the same export request, when the data export type is to send data to the corresponding target device in a communication manner, a corresponding communication-based export task is established. In the form of sending data to the target device in a communication manner, the data receiving object is the target device, and the associated interface of the data receiving object correspondingly includes the communication interface of the target device. After the data management system receives the export request, it creates a corresponding export task. If the export state machine currently does not contain an export task that is being executed and not completed, the data management system automatically executes the entire export state machine and calls the communication interface to send the export data to the target device. After the target device receives the export data, it creates and writes data to the target format file by itself, and closes the file after the export is completed.

[0063] Please refer to Figure 4 , in an optional example, the export execution process of the communication-based export task includes: S21, hooking the communication interface to the data management system; S22, the communication interface queries the export request instruction through the receiving function, and submits the export request through the request interface according to the information included in the export request instruction; S23, creating an export task according to the export request, executing the export state machine, and calling the communication interface to send the export data that meets the export request to the target device during the process; S24, after the target device receives the export data, it creates and writes the data to the target format file by itself, and closes the file after the export is completed. Among them, the target format file can be a csv file. After the export is completed, the user can open the csv file on the target device side to view the export data. Optionally, the data management system is an export system that executes an independent data export function.

[0064] In the above embodiment, the data management system is used to provide a data export service, which is not limited to a specific form of data export type. When the export request includes a data export type of sending data to the corresponding target device in a communication manner, the communication interface of the target device is hooked to the data management system to achieve data export in a communication manner, so as to meet and not be limited to the data export requirements in a communication manner. Among them, in an export request, the target device can be one or more, which are distinguished by the target device identifier. In the communication-based export task, the hooking of the communication interfaces of different target devices can be distinguished according to the target device identifier, and the export data can be sent to different target devices respectively.

[0065] In some embodiments, S101, receiving an export request and adding the export request to the export request queue includes:

[0066] Querying the export request instruction through the communication interface, and submitting the export request with the requested information through the request interface according to the export request instruction;

[0067] Add the export request to the export request queue.

[0068] Among them, the data management system can query the export request instruction through the receive function in the communication interface. Optionally, the export request instruction can be a preset instruction for characterizing that the user needs to export data, and the form of the export request instruction can be a voice instruction or a gesture instruction received through the attached communication interface, etc. After the data management system receives the export request instruction and submits the export request, add the export request to the export request queue.

[0069] In the above embodiment, the data management system attaches to the communication interface of the target device, queries the export request instruction through the communication interface, and submits the export request. By attaching the communication interface of the target device to the data management system, it is possible to receive the export request instruction and submit the export request regardless of the communication method.

[0070] In some embodiments, the execution export state machine includes:

[0071] Determine the export data according to the export task, and determine the header content from the export data according to the configuration format information;

[0072] Export the header content in segments with a fixed number of columns until the header content is completely exported;

[0073] Determine the table content from the export data according to the configuration format information;

[0074] Export the table content in segments with a fixed number of columns until the table content is completely exported.

[0075] The data management system provides a data export operation interface, and through the data export operation interface, multiple configuration items are provided. Each configuration item can be selected by the user to set corresponding configuration information. Each configuration item can be set according to actual requirements. In this embodiment, the configuration items include a configuration item for setting the recording carrier of the exported data. For example, the configuration item includes options for the optional file types to which the exported data is written, so as to provide the user with the option to select the recording carrier of the exported data. Generally, different types of exported data are suitable for different recording carriers. Optionally, under the option of selecting a certain recording carrier, the configuration item can further include a configuration item for the data format for recording data under this recording carrier. For example, under the option of selecting an excel file as the recording carrier, the configuration item includes format setting options for the header content and the table content. The data management system obtains configuration format information corresponding to the export request according to the configuration operations on the configuration items in the data export operation interface. The header content is exported in segments with a fixed number of columns, and the number of columns for the segmented export of the header content can be determined through the configuration format information. For example, it can be set through the format setting options for the header content. The table content is exported in segments with a fixed number of columns, and the number of columns for the segmented export of the table content can also be determined through the configuration format information. For example, it can be set through the format setting options for the table content.

[0076] In the above embodiment, the exported data can be a log record. For the export of the log record, an excel file can be set as the recording carrier. Determine the range of log record data to be exported for the current export task. First, export the header content in segments according to the configuration format information. The header has only one row. After the export is completed, switch to the export of the table content and export the table content in segments. The data management system performs segmented formatting processing on the exported data and supports format configuration of the exported data, which can mask the length change of the exported data. Through segmented loading, it is ensured that the size of the cache used for data export does not increase with the increase in the length of the exported data.

[0077] Optionally, the exported data is a log record that is split into data excerpt information and data parts for storage. The data excerpt information includes an index information byte, an identification information byte, and an address information byte. The index information byte is used to record the index information of the log record. The identification information byte is used to record the format processing operation type of the log record. The address information byte is used to write the storage address of the log record;

[0078] The step of exporting the table content in segments with a fixed number of columns until the export of the table content is completed includes:

[0079] Filter according to the index information in the data excerpt information of the log record to determine the target log record that meets the export request;

[0080] Load the corresponding data part according to the storage address in the data extraction information of the target log record, and restore the data part to the original data according to the format processing operation type in the data extraction information;

[0081] Convert the original data into a table content string, and segment and export the table content string with a fixed number of columns until the table content is exported completely.

[0082] The exported data can be the recorded data formed during the operation of different business systems. For example, for a certain work platform software, the recorded data during the operation includes operation log records such as the login, operation, permission assignment, and permission deletion of authorized users during use. Then the exported data can be the operation diary record of a certain authorized user of the work platform software within a certain time period, or the operation log records of all authorized users within a certain time period, etc.; Another example is that for a certain industrial software, the recorded data during the operation mainly includes work log records such as work status, faults, and interruptions during use. Then the exported data can be the work log record of the industrial software within a certain time period, or the work log record with a fault work status of the work software within a certain time period, etc.

[0083] In this embodiment, the data management system can also obtain the recorded data generated during the operation of the business system, format the recorded data, and store the recorded data separately into an information part and a data part independently to implement the storage management of the recorded data. The information part is stored using structured data extraction information, and the data extraction information is set to include an index information byte, an identification information byte, and an address information byte. The index information byte is used to record the index information corresponding to the recorded data, the identification information byte is used to record the format processing operation type corresponding to the recorded data, and the address information byte is used to write the storage address corresponding to the recorded data. Store the recorded data after the format processing operation executed by the data management platform into the storage address written in the address information byte.

[0084] During the process of the data management system implementing the storage management of the recorded data, the format processing operation type performed on the recorded data can refer to the processing operation type of format conversion according to whether the length of the recorded data exceeds a certain value. For example, when the length of a single recorded data exceeds a certain value, a differential compression processing operation is performed. The storage address of the recorded data can refer to any storage space linked according to the user's selection operation of the data storage address. For example, a certain partition in the current electronic device can be selected as the storage address of the recorded data; it can also refer to the storage space linked and determined according to the user's configuration of the data storage rule. For example, when the user configures the current electronic device to insert an external storage device, the external storage device is preferentially used as the storage address of the recorded data.

[0085] Please refer to Figure 5 , which is a schematic diagram of the structured data extraction information in an optional example. Among them, the structure of the data extraction information adopts the defined T-D structure, with 32 bits for both the index information byte and the address information byte, and 1 bit for the identification information byte. The index information byte is used to record the index information, and the address information byte records the data storage address. For the log records split and stored independently as data extraction information and the data part respectively, on the one hand, the storage spaces can be independent of each other, realizing the independence of the data storage address from the data generation, lifting the restriction on sequential data storage, and effectively improving the utilization rate of the storage space; on the other hand, the recording process of the data can be decoupled from the recording medium, reducing the dependence of data storage on the recording medium and having higher reliability; on the other hand, using the structured data extraction information can conveniently implement management operations such as fast query, screening, and processing, and effectively improve the data management performance.

[0086] Taking the log record of recording data as an example, the data extraction information of the log record includes extracting specific fields from the original data of the log record to form index information and filling it into the index information byte accordingly; judging whether to differentially compress it according to the length of a single log record according to a preset rule to determine the value of the flag bit in the identification information byte. For example, when differential compression has been performed on the recorded data in the identification information byte, the corresponding flag bit is set to 1, and when differential compression has not been performed, the corresponding flag bit is set to 0; the starting offset address of the corresponding log record is written into the address information byte. If the log record is a periodic log record, the index information can correspondingly be the timestamp corresponding to the log record; if the log record is an event-based log record, the index information can correspondingly be the event label.

[0087] In an optional example, please refer to Figure 6, the exported data is a log record. The export of table content includes: S31, filtering according to the index information in the data extraction information to filter out the target log records that meet the export request. For example, for the export mode of serial number range, calculate the target export record serial number range according to the export request, and lock the log records within the target export record serial number range as the log records that meet the export request. For the export mode of date range, calculate the target export record date range according to the export request, load the timestamp of the record by loading the index information byte of the record, and only the records within the target export record date range are hit and passed the filter. S32, according to the address information byte of the data extraction information of the target log record, load the data part of the target log record from the corresponding storage address, and restore the data part to the original data according to the value of the identification bit in the identification information byte; S33, convert the original data into a table content string according to the configuration of the export data format. The data management system exports the table content string in fixed column numbers, and calls an external write interface to write the table content string into the target device; S34, the data management system will judge whether all target records have been exported during each execution process. If there are unexported log records, continue to execute the current export process; otherwise, remove the currently executing export request from the export execution queue.

[0088] In the above embodiment, the export function of the data management system is completed in two stages: the export of the table header content and the export of the table content. The export of the table header content and the table content are respectively exported in fixed column numbers. For the log records in which the data extraction information and the data part are stored separately, specifically, for the log records with time or serial number as the index information, the target log records that meet the export request can be filtered by two filtering methods: time range or serial number range. During the filtering process, the index information of the log records can be loaded separately. First, filter the matching target log records by loading the data extraction information, and then load the data part of the corresponding target log record and convert and export it according to the configured export format, which can reduce the processing volume of data export and improve the efficiency of data export.

[0089] Optionally, sending the exported data to the data receiving object through the associated interface of the corresponding data receiving object hooked to the data management system includes:

[0090] Call the creation interface of the data receiving object to create an export file;

[0091] Call the write interface of the data receiving object, and according to the device identifier of the data receiving object, export and write the table header content and the table content into the export file of the corresponding data receiving object.

[0092] The data receiving object can be other target devices independent of the device where the data management system is located. The export request contains information about the data receiving object, such as the device ID (Identity document, label) of one or more target devices. The data management system can distinguish according to the device ID of the target device, call the creation interface of the file system of the corresponding target device to create an export file, and after sending the export data to the target device through the associated interface of each target device, call the write interface of the file system of the corresponding target device to write the export data into the export file.

[0093] In the above embodiment, the data management system can support the data export to multiple target devices for the same export request by calling an external execution interface, and is not limited to the file format for data export. It can be applied to the data export requirements in more application scenarios and improve the export efficiency.

[0094] Optionally, the execution export state machine further includes:

[0095] Determine that the current export task is completed, and switch the export state machine to the idle state; the export state machine being in the idle state indicates that there is currently no ongoing and unfinished export task.

[0096] When the data export corresponding to each completed export request ends, set the export state machine to the idle state as an indication that an export request is completed. The export state machine being in the idle state means that there is currently no ongoing and unfinished export task in the data management system. This can conveniently and accurately identify the situation where the export execution queue is empty while the export state machine is executing an export task, avoiding errors caused by task conflicts.

[0097] To have a more comprehensive understanding of the data management method provided by the embodiments of the present application, please refer to Figure 7 , and the following takes the export data as the log record of the embedded system as an example for illustration. The data management method includes:

[0098] S41. Define two circular queues A and B in the data management system. Circular queue A is the export request queue, and circular queue B is the export execution queue;

[0099] S42. The data management system provides a request interface to receive an export request and add the received export request to circular queue A;

[0100] S43. The export service occupies a single thread and continuously checks whether circular queue A is empty. When circular queue A is not empty and circular queue B is empty, dequeue the first export request m in A and add it to circular queue B;

[0101] S44. The export service continuously checks whether the circular queue B is empty. If the circular queue B is empty, this execution is exited; if the circular queue is not empty and the export state machine is currently in the idle state, the export request m in the circular queue B is taken out, and according to the configuration in the export request m, the associated interface that is hooked is called to create an export file, and the export data that meets the export request m is used as the input information of the export state machine.

[0102] S45. The export service executes the export state machine.

[0103] An export task corresponding to the export request m is established. Please refer to Figure 8 , the export state machine is in the idle state; when there is a new export task, the header content in the export data is determined according to the user's configuration of the export data format, and the header content is exported in segments until the export of the header content is completed; the table content in the export data is determined according to the user's configuration of the export data format, and the table content is exported in segments until the export of the table content is completed; when it is determined that the export is completed, the export file is closed; the export state machine is switched to the idle state again.

[0104] S46. The export service execution is completed; return to S43.

[0105] The data management method provided by the above embodiment has at least the following characteristics:

[0106] First, the data management system is implemented through two circular queues and an export state machine, has no strict requirements on the implementation platform, and both are standard data structures and design methods, so the data management method has strong portability;

[0107] Second, the data export is executed through the hooked interface, and the export process is abstracted in the way of interfaces (open, write, close), without being limited to whether the export method is file type or communication type;

[0108] Third, the export state machine realizes the segmented loading of the export data, so as to ensure that the size of the export cache does not increase with the increase of the data length, and shields the change of the data length through the internal segmentation method of the data management system, simplifying the implementation work outside the data management system;

[0109] Fourth, through circular queues A and B, the request management of the export service is realized, avoiding the loss of export requests during the export process;

[0110] Fifth, for the log records of the embedded system, it supports the export of two screening methods: time range and serial number range, and also supports the export of log records in the append mode. The log records are split into two independent parts: data extraction information and data for storage. Therefore, the screening process can be carried out by separately loading the index information of the log records, improving the export performance.

[0111] Please refer to Figure 9 Figure 9 , on the other hand, this application also provides a data management system, including a request queue module 121 for receiving an export request and adding the export request to an export request queue; an execution queue module 122 for adding the export request in the export request queue to an export execution queue; an export service module 123 for, if the export execution queue is not empty and currently does not contain an executing and unfinished export task, taking out the corresponding export request from the export execution queue and establishing a corresponding export task according to the data export type supported by the data receiving object included in the export request; and an export execution module 124 for executing an export state machine according to the export task and sending the export data to the data receiving object through an associated interface of the corresponding data receiving object hooked to the data management system.

[0112] Optionally, the export execution module 124 is specifically configured to hook an open interface, a write interface, and a close interface of a file system to the data management system; if the export task is a file type export task, execute the export state machine according to the file type export task, and call the open interface, the write interface, and the close interface to write the export data into a target file.

[0113] Optionally, the request queue module 121 is specifically configured to obtain export configuration information through a request interface according to an export configuration operation, submit an export request based on the export configuration information; and add the export request to the export request queue.

[0114] Optionally, the export execution module 124 is specifically configured to hook a communication interface to the data management system; if the export task is a communication type export task, execute the export state machine according to the communication type export task, and call the communication interface to send the export data to a target device, and the target device receives the export data and writes it into a target file.

[0115] Optionally, the request queue module 121 is specifically configured to query an export request instruction through the communication interface, submit the requested information as an export request through the request interface according to the export request instruction; and add the export request to the export request queue.

[0116] Optionally, the export execution module is further configured to determine export data according to the export task, determine header content from the export data according to configuration format information; segment and export the header content in a fixed number of columns until the header content is completely exported; determine table content from the export data according to the configuration format information; and segment and export the table content in a fixed number of columns until the table content is completely exported.

[0117] Optionally, the exported data is a log record that is split into data excerpt information and a data part for storage. The data excerpt information includes an index information byte, an identification information byte, and an address information byte. The index information byte is used to record the index information of the log record. The identification information byte is used to record the format processing operation type of the log record. The address information byte is used to write the storage address of the log record.

[0118] The export execution module 124 is further configured to filter according to the index information in the data excerpt information of the log record to determine a target log record that meets the export request; load the corresponding data part according to the storage address in the data excerpt information of the target log record, and restore the data part to the original data according to the format processing operation type in the data excerpt information; convert the original data into a table content string, and export the table content string in segments with a fixed number of columns until the table content is exported.

[0119] Optionally, the export execution module 124 is further configured to call the creation interface of the data receiving object to create an export file; call the write interface of the data receiving object, and according to the device identification of the data receiving object, export and write the header content and the table content into the export file corresponding to the data receiving object.

[0120] Optionally, the export execution module 124 is further configured to determine that the current export task is completed and switch the export state machine to the idle state; the export state machine being in the idle state indicates that there is currently no ongoing and unfinished export task.

[0121] It should be noted that the structure of the data management system provided in the embodiments of the present application does not limit the data management system. Each of the above modules can be implemented in whole or in part by software, hardware, and their combinations. The above modules can be embedded in the processor in the computer device in hardware form or independent of it, or stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules. In other embodiments, the data management system may include more or fewer modules than shown in the figure.

[0122] Please refer to Figure 10 , another aspect of the embodiments of the present application provides a data management device, including a memory 211 and a processor 212. The memory 211 stores a computer program. When the computer program is executed by the processor, the processor 212 executes the steps of the data management method provided in any one of the above embodiments of the present application.

[0123] In another aspect of the embodiments of the present application, there is provided a computer program product. When the instructions in the computer program product are executed by a processor of a data management device, the data management device is caused to perform the steps of the data management method provided in any one of the above embodiments of the present application.

[0124] In another aspect of the embodiments of the present application, there is also provided a storage medium storing a computer program. When the computer program is executed by a processor, the processor is caused to perform the steps of the control method of the cooling system provided in any one of the above embodiments of the present application.

[0125] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods provided in the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0126] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. The protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A data management method, applied to a data management system, characterized in that, Including: Receiving an export request and adding the export request to an export request queue; Adding the export request in the export request queue to an export execution queue; If the export execution queue is not empty and currently does not contain an executing and unfinished export task, taking out the corresponding export request from the export execution queue, and establishing an export task corresponding to each data export type according to multiple data receiving objects included in the export request that support different data export types; According to the export task, executing an export state machine, and sending export data to the data receiving object through an associated interface of the data receiving object hooked to the data management system.

2. The data management method according to claim 1, wherein The step of, according to the export task, executing an export state machine and sending export data to the data receiving object through an associated interface of the data receiving object hooked to the data management system includes: Hooking an open interface, a write interface, and a close interface of a file system to the data management system; If the export task is a file-type export task, according to the file-type export task, executing an export state machine, and calling the open interface, the write interface, and the close interface to write the export data into a target file.

3. The data management method according to claim 2, wherein The step of receiving an export request and adding the export request to an export request queue includes: Obtaining export configuration information through a request interface according to an export configuration operation, and submitting an export request based on the export configuration information; Adding the export request to the export request queue.

4. The data management method according to claim 1, wherein The step of, according to the export task, executing an export state machine and sending export data to the data receiving object through an associated interface of the data receiving object hooked to the data management system includes: Hooking a communication interface to the data management system; If the export task is a communication-type export task, according to the communication-type export task, executing an export state machine, and calling the communication interface to send the export data to a target device, and the target device receives the export data and writes it into a target file.

5. The data management method according to claim 4, characterized in that, The step of receiving an export request and adding the export request to an export request queue includes: Querying an export request instruction through the communication interface, and submitting the requested information as an export request through a request interface according to the export request instruction; Adding the export request to the export request queue.

6. The data management method according to claim 1, wherein The step of executing the export state machine includes: Determining export data according to the export task, and determining header content from the export data according to configuration format information; Exporting the header content in segments with a fixed number of columns until the header content is completely exported; Determining table content from the export data according to the configuration format information; Exporting the table content in segments with a fixed number of columns until the table content is completely exported.

7. The data management method according to claim 6, wherein The export data is a log record stored by splitting it into data extraction information and a data part. The data extraction information includes an index information byte, an identification information byte, and an address information byte. The index information byte is used to record the index information of the log record, the identification information byte is used to record the format processing operation type of the log record, and the address information byte is used to write the storage address of the log record; Segmenting and exporting the table content by a fixed number of columns until the export of the table content is completed, includes: Filtering according to the index information in the data extraction information of the log record to determine the target log record that meets the export request; Loading the corresponding data part according to the storage address in the data extraction information of the target log record, and restoring the data part to the original data according to the format processing operation type in the data extraction information; Converting the original data into a table content string, and segmenting and exporting the table content string by a fixed number of columns until the export of the table content is completed.

8. The data management method according to claim 6, wherein Sending the exported data to the data receiving object through the associated interface hooked to the corresponding data receiving object of the data management system, includes: Invoking the creation interface of the data receiving object to create an export file; Invoking the write interface of the data receiving object, and exporting and writing the header content and the table content into the export file of the corresponding data receiving object according to the device identifier of the data receiving object.

9. The data management method according to claim 6, wherein The execution of the export state machine further includes: Determining that the current export task is completed, and switching the export state machine to the idle state; the export state machine being in the idle state indicates that there is currently no ongoing and unfinished export task.

10. A data management system, characterized in that, Includes: A request queue module for receiving an export request and adding the export request to an export request queue; An execution queue module for adding the export request in the export request queue to an export execution queue; An export service module for, if the export execution queue is not empty and there is currently no ongoing and unfinished export task, taking out the corresponding export request from the export execution queue, and establishing an export task corresponding to each data export type according to multiple data receiving objects supporting different data export types included in the export request; An export execution module for executing the export state machine according to the export task, and sending the exported data to the data receiving object through the associated interface hooked to the corresponding data receiving object of the data management system.

11. A data management device, characterized in that, Includes a memory and a processor, where the memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to execute the steps of the data management method according to any one of claims 1 to 9.

12. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the data management device, the data management device is caused to execute the steps of the data management method according to any one of claims 1 to 9.

13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the processor is caused to execute the steps of the data management method according to any one of claims 1 to 9.

Images