Real-time data storage and recovery method and device for cloud order information, and storage medium
By saving SQL statements to a file and using memory mapping technology, the database execution efficiency bottleneck is solved, and data recovery is achieved in high-precision, fast response and persistence scenarios are achieved, meeting real-time requirements.
Patent Information
- Application Number
- CN202410138118.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-31
- Publication Date
- 2025-07-04
AI Technical Summary
The existing data storage methods rely on databases, resulting in the inability to meet real-time requirements in high-precision, fast response and persistence scenarios, and there is a bottleneck in database execution efficiency.
By saving the SQL statements to be executed first to the file, reading and writing in memory map, combining asynchronous saving threads and the first-in-first-out principle, recording file name and offset information, the data is quickly restored and persisted.
It avoids the bottleneck problem of database execution efficiency, and can quickly restore unsaved data after the program is restarted, meeting the application needs of high-precision, fast response and persistence.
Smart Images
Figure CN120256441A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and particularly to a method, device, and storage medium for real-time data storage and recovery of cloud order information. Background Art
[0002] With the development of informatization, the amount of data in various industries is increasing, and there are scenarios where a large amount of data is stored in a database. Due to various abnormal situations, the program restarts, and corresponding problems of data storage and data recovery are brought about during the data processing process.
[0003] The existing data storage methods mainly rely on databases, but due to the bottleneck of the database execution efficiency, they cannot meet the real-time requirements. Summary of the Invention
[0004] In view of this, it is necessary to solve the shortcomings existing in the prior art. At the same time, in the scenario of meeting high-precision, fast response, and persistence of data, ensure the accuracy and persistence of data, and realize the rapid execution and recovery of data, and provide a method, system, device, and storage medium for real-time data storage and recovery of cloud order information.
[0005] The present invention is achieved through the following technical solutions:
[0006] A method for real-time data storage and recovery of cloud order information includes the following steps:
[0007] S1: A user places an order through a client to generate an order instruction;
[0008] S2: Upload the first order information generated by the order instruction to a server, and the server processes the first order information through a real-time data storage and recovery mechanism to generate second order information;
[0009] S3: The server sends the second order information to the client, and the client displays the second order information returned by the server and realizes operations including modification and deletion of the second order information.
[0010] Preferably, the real-time data storage and recovery mechanism includes:
[0011] When the program of the real-time data storage and recovery mechanism starts, a thread continuously scans a pre-established container, that is, scans the elements in the container according to the first order information. If the container includes the element, execute the first SQL statement referred to by the element and save the data in the database.
[0012] Preferably, the real-time data storage and recovery mechanism further includes:
[0013] During the execution of the program of the real-time data storage and recovery mechanism, when there is a new SQL statement to be executed, store the SQL statement in a pre-established file, and record the file name and offset information of the file in the configuration file;
[0014] Save the SQL statement into the container.
[0015] Preferably, the container further includes:
[0016] The container adopts the first-in, first-out principle, generates the SQL statement according to the memory operation to be executed, and adds the SQL statement to the end of the container, that is, the elements in the container are the mapping of the program's memory operations.
[0017] Preferably, the thread further includes:
[0018] The thread is an asynchronous save thread, and the memory operation mapping executed by the asynchronous save thread is not synchronized with the actual memory operation. The memory operation mapping and the actual memory operation work in a producer-consumer mode;
[0019] Among them, the actual memory operation is the producer, adding the SQL statement to the container is consumed by the consumer, and the consumer is the current thread, that is, the current thread continuously scans the container.
[0020] Preferably, the file further includes:
[0021] The file reads and writes the SQL statement in a memory mapping manner for improving the reading and writing speed of the file;
[0022] Process the file when the file reaches the set number or after a certain period of time;
[0023] Among them, processing the file means that when the file reaches the set number or after a certain period of time, perform an operation of file splitting for managing the amount of stored data and maintaining the readability of the file. The file splitting is to create a new file for storing subsequent SQL statements.
[0024] Preferably, during the scanning process, when encountering a program restart, according to the offset recorded in the configuration file, continue to read and execute the second SQL statement in the SQL statements that should have been executed but actually not executed last time, that is, read the data with the specified length in the configuration file item by item according to the length of the record header, and restore the data that has not been saved to the database.
[0025] Preferably, the second SQL statement further includes:
[0026] The execution of the second SQL statement is independent of the container and is blocked and continuously executed before the program initialization. Then, the program is initialized and the service is restarted to ensure that the data that was not saved before can be restored after the program restarts and the subsequent operations can continue to be executed.
[0027] A computer device, comprising a memory and a processor, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the real-time data storage and recovery method for cloud order information according to an embodiment of the present invention.
[0028] A storage medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the real-time data storage and recovery method for cloud order information according to an embodiment of the present invention.
[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0030] 1) By first saving the SQL statements to be executed into a file and then executing them one by one, and reading and writing in a memory-mapped manner, the present invention avoids the bottleneck problem of database execution efficiency.
[0031] 2) By recording the file name and offset of the execution, the present invention can quickly restore the unsaved data after the program restarts. This method has significant advantages in application scenarios that require high precision, fast response, and persistence. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention.
[0033] Figure 1 It is a schematic flow chart of the real-time data storage and recovery method for cloud order information of the present invention;
[0034] Figure 2 It is a schematic flow chart of the real-time data storage and recovery mechanism of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0035] To make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.
[0036] Those skilled in the art of the present technology can understand that, unless specifically stated otherwise, the singular forms "a", "an", and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present invention means the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups.
[0037] Embodiment 1
[0038] Please refer to Figure 1 As shown, the real-time data storage and recovery method for cloud order information provided in this embodiment includes the following steps:
[0039] S1: The user places an order through the client to generate an order instruction;
[0040] S2: Upload the first order information generated by the order instruction to the server, and the server processes the first order information through the real-time data storage and recovery mechanism to generate the second order information;
[0041] S3: The server sends the second order information to the client, and the client displays the second order information returned by the server and realizes operations including modification and deletion of the second order information.
[0042] Preferably, the real-time data storage and recovery mechanism includes:
[0043] When the program of the real-time data storage and recovery mechanism starts, the thread continuously scans the pre-established container, that is, scans the elements in the container according to the first order information. If the container includes the element, execute the first SQL statement referred to by the element and save the data in the database. Specifically, in this embodiment, the pre-established container (Vector) can be a data structure, and the SQL statements include, but are not limited to, operations such as adding, deleting, modifying, and querying the user status and business status.
[0044] Preferably, if the container contains elements, the database instructions referred to in the elements are executed one by one, so as to save the data to the database. After executing one statement, record the file name and offset corresponding to the statement and save them in the configuration file, which is the "actually executed SQL statement". This method can be imagined as similar to the cursor movement method when printing, and the execution of the above "SQL statement record to be executed" is completed line by line. What the program records is actually the current position information of the cursor (the so-called "offset"). This process is mainly used to continue to execute the unfinished SQL statement operations after the program restarts.
[0045] Preferably, the real-time data storage and recovery mechanism further includes:
[0046] During the program execution of the real-time data storage and recovery mechanism, when there is a new SQL statement to be executed, store the SQL statement in a pre-established file, and record the file name and offset information of the file to the configuration file;
[0047] Save the SQL statement to the container. Specifically, in this embodiment, please refer to Figure 2 As shown, when the program starts, generate the SQL statement to be executed, save the SQL statement to a file, record the file name and offset, use the memory mapping method to accelerate the saving, quickly process the data, save the SQL statement to the container, and sequentially execute the SQL statements in the container, and store the data obtained from the executed SQL statements in the database.
[0048] Preferably, in order to track the executed SQL statements, the program introduces a file to store the executed SQL statements, and this file can be created in the way of LINUX I / O.
[0049] Among them, in Linux, the following multiple I / O (input / output) methods can be used to create files. I / O can be divided into high-level I / O and low-level I / O. High-level I / O is usually also called buffered I / O, such as the standard I / O library provided by the ANSI C library. Low-level I / O is usually also called unbuffered I / O, which is a system call provided by Linux, such as open, read, write, etc. Buffered I / O adopts a certain strategy before the system call, is slow, but safer than unbuffered I / O, such as fopen, fread, fwrite, etc.:
[0050] (1) Use the command line tool touch:
[0051] touch ex_file
[0052] This will create a blank file named ex_file in the current directory. If the file already exists, its access time will be updated.
[0053] (2) Write a program in C language for file operations;
[0054]
[0055] Open or create a file named ex_file through the fopen function and specify the "w" parameter to open the file in write mode. If the file cannot be created, NULL will be returned; otherwise, print a prompt message and then close the file.
[0056] (3) Write a program in Python for file operations:
[0057] try:
[0058] file = open('ex_file', 'w') # Open or create a file named ex_file and set it to write mode
[0059] except Exception as e:
[0060] print("Unable to create file")
[0061] else:
[0062] print("Successfully created file")
[0063] finally:
[0064] file.close() # Close the file
[0065] Open or create a file named ex_file through the built-in open function and specify the 'w' parameter to open the file in write mode. If the file cannot be created, an exception will be thrown; otherwise, output a prompt message and then close the file.
[0066] In this embodiment, the C language method is adopted to create a file. By adopting the memory mapping method to create a file, the bottleneck problem of the database execution efficiency is avoided. In other embodiments, other methods can be adopted for creation, and this embodiment does not limit the way of the file.
[0067] Preferably, the method of compiling the configuration file is to save the length at the head of a record, so that when reading, the specified length can be read according to the length saved at the head, and then continue to read. Compare the SQL statements to be executed with the actually executed SQL statements, so as to know which statements have not been executed, and then the execution can be completed. The two storage methods of saving in files and configuration files are used to ensure data persistence and easy management.
[0068] Preferably, the container further includes:
[0069] The container adopts the first-in-first-out principle, generates the SQL statement according to the memory operation to be executed, and adds the SQL statement to the end of the container. That is, the elements in the container are the mappings of the program's memory operations. Specifically, in this embodiment, the SQL statement is added to the end of the container. When executing the SQL statements in the container, the SQL statements that enter the container first are processed first, and the SQL statements that enter later will be postponed. Each element in the container is a mapping of the program's memory operation, which is used to achieve the persistence of the memory state, that is, the state that can be resumed and continued to execute last after the server restarts.
[0070] Preferably, the thread further includes:
[0071] The thread is an asynchronous save thread, and the memory operation mapping executed by the asynchronous save thread is not synchronized with the actual memory operation. The memory operation mapping and the actual memory operation work in a producer-consumer mode;
[0072] Among them, the actual memory operation is the producer, and adding the SQL statement to the container is consumed by the consumer. The consumer is the current thread, that is, the current thread continuously scans the container.
[0073] Preferably, the file further includes:
[0074] The file reads and writes the SQL statement by means of memory mapping used to improve the reading and writing speed of the file;
[0075] The file processes the file when it reaches the set number or after a certain period of time;
[0076] Among them, processing the file means that when the file reaches the set number or after a certain period of time, an operation of file splitting is performed to manage the amount of stored data and maintain the readability of the file. The file splitting means creating a new file to store subsequent SQL statements.
[0077] Preferably, when the content in the file exceeds 10,000 items, a new file is created to store subsequent SQL statement records; when one hour has passed, a new file is created to store subsequent SQL statement records. Among them, for the two situations where the content exceeds 10,000 items and one hour has passed, file splitting is performed based on the earlier occurrence.
[0078] Preferably, during the scanning process, when a program restart is encountered, according to the offset recorded in the configuration file, continue to read and execute the second SQL statement in the SQL statements that should have been executed but were not actually executed last time, that is, read data of a specified length in the configuration file item by item according to the length of the record header, and restore the data that has not been saved to the database. Specifically, in this embodiment, after restart, upon receiving a restoration instruction, the restoration instruction indicates the configuration file. According to the offset and file name recorded in the configuration file, find the file of the SQL statement that was executed last time, and based on the file, find the SQL statements that should have been executed but were not actually executed in the container last time to restore the data that has not been saved to the database.
[0079] Preferably, the second SQL statement further includes:
[0080] The execution of the second SQL statement is independent of the container and is continuously executed in a blocking manner before program initialization, and then program initialization is performed and the service is restarted to ensure that after the program restarts, the previously unsaved data can be restored and subsequent operations can continue to be executed.
[0081] This embodiment also provides a real-time data storage and recovery system, which is characterized by including:
[0082] A data storage module, used to save data to the database and add SQL statements to the execution queue of the container;
[0083] A data execution module, used to sequentially execute the SQL statements in the execution queue in an opened thread, store the executed SQL statements in the file of the data storage module, and record the file name and offset in the configuration file of the data storage module;
[0084] A data recovery module, used in the case of program restart, the data recovery module reads the last executed file and the offset from the configuration file, and then reads the remaining unexecuted SQL statements from the container according to the file and the offset, and executes them to restore the data in the database.
[0085] Embodiment 2
[0086] In this embodiment, a computer device is proposed, which includes a memory and a processor. Computer-readable instructions are stored in the memory. When the computer-readable instructions are executed by the processor, the processor is caused to execute the steps of the real-time data storage and recovery method of the cloud order information described in Embodiment 1.
[0087] In this embodiment, a storage medium storing computer-readable instructions is proposed. The computer-readable instructions, when executed by one or more processors, cause the one or more processors to execute the steps of the real-time data storage and recovery method of the cloud order information described in Embodiment 1.
[0088] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. This program can be stored in a computer-readable storage medium. The storage medium can include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.
[0089] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered that the scope recorded in this specification.
[0090] The above is only the preferred embodiment of the present invention. The protection scope of the present invention is not limited to the above embodiments. Any technical solution falling within the idea of the present invention belongs to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, several improvements and refinements made without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. A real-time data storage and recovery method for cloud order information, characterized in that, It includes the following steps: S1: The user places an order through the client to generate an order instruction; S2: Upload the first order information generated by the order instruction to the server, and the server processes the first order information through a real-time data storage and recovery mechanism to generate second order information; S3: The server sends the second order information to the client, and the client displays the second order information returned by the server and realizes operations including modification and deletion of the second order information.
2. The real-time data storage and recovery method for cloud order information according to claim 1, characterized in that, The real-time data storage and recovery mechanism includes: When the program of the real-time data storage and recovery mechanism starts, the thread continuously scans a pre-established container, that is, scans the elements in the container according to the first order information. If the container includes the element, execute the first SQL statement referred to by the element and save the data in the database.
3. The real-time data storage and recovery method for cloud order information according to claim 2, wherein, The real-time data storage and recovery mechanism further includes: During the execution of the program of the real-time data storage and recovery mechanism, when there is a new SQL statement to be executed, store the SQL statement in a pre-established file, and record the file name and offset information of the file in the configuration file; Save the SQL statement to the container.
4. The real-time data storage and recovery method for cloud order information according to claim 3, characterized in that The container further includes: The container adopts the first-in-first-out principle, generates the SQL statement according to the memory operation to be executed, and adds the SQL statement to the end of the container, that is, the element in the container is the mapping of the program's memory operation.
5. The real-time data storage and recovery method for cloud order information according to claim 4, wherein The thread further includes: The thread is an asynchronous save thread, and the memory operation mapping executed by the asynchronous save thread is not synchronized with the actual memory operation. The memory operation mapping and the actual memory operation work in a producer-consumer mode; Among them, the actual memory operation is the producer, adding the SQL statement to the container is consumed by the consumer, and the consumer is the current thread, that is, the current thread continuously scans the container.
6. The real-time data storage and recovery method for cloud order information according to claim 5, characterized in that, The file further includes: The file reads and writes the SQL statement by means of memory mapping used to improve the read and write speed of the file; Process the file when the file reaches the set number or after a certain period of time; Among them, processing the file means that when the file reaches the set number or after a certain period of time, perform an operation of file splitting for managing the amount of stored data and maintaining the readability of the file. The file splitting means creating a new file to store subsequent SQL statements.
7. The real-time data storage and recovery method for cloud order information according to claim 6, characterized in that, During the scanning process, when the program restarts, according to the offset recorded in the configuration file, continue to read and execute the second SQL statement in the SQL statement that should have been executed but was not actually executed last time, that is, read the data of the specified length in the configuration file item by item according to the length of the record header, and restore the data not saved to the database.
8. The real-time data storage and recovery method for cloud order information according to claim 7, characterized in that The second SQL statement further includes: The execution of the second SQL statement is independent of the container and is blocked from continuous execution before program initialization. Then, the program is initialized and the service is restarted to ensure that the data that was not saved previously can be restored after the program restarts and subsequent operations can continue to be executed.
9. A computer device, characterized in that, It includes a memory and a processor. Computer-readable instructions are stored in the memory. When the computer-readable instructions are executed by the processor, the processor is caused to execute the steps of the real-time data storage and recovery method for cloud order information according to any one of claims 1 to 8.
10. A storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to execute the steps of the real-time data storage and recovery method for cloud order information according to any one of claims 1 to 8.