File-based quick storage retrieval method, device and equipment and storage medium

Abstract

The invention provides a file-based quick storage retrieval method, device and equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring to-be-recorded data; processing the to-be-recorded data into a plurality of compressed character strings with retrieval keys; creating a cache file, writing the plurality of compressed character strings into the cache file, and recording a position recording pointer of each compressed character string; and associatively storing the retrieval key and the position recording pointer of each compressed character string into a memory. According to the method, high efficiency of file storage and rapidity of memory retrieval can be combined, on one hand, storage requirements can be reduced by compressing data, and the storage efficiency is improved; and on the other hand, the index is established in the memory, so that the data retrieval speed can be increased.

Description

Background Art

[0002] With the advent of the intelligent era, new technologies represented by AI (Artificial Intelligence) and big data technologies have flourished, leading to an explosive growth in the scale of information communication networks that carry information flows. In the end-to-end management of the network, the routing cascade algorithm is one of the important technologies.

[0003] In the related art, in the routing concatenation algorithm, the basic data used for routing concatenation is usually first loaded into the memory, and then the routing concatenation is performed. As the amount of data of end-to-end business objects becomes larger and larger, the data loading method in the related art will lead to memory expansion problems.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a file-based fast storage and retrieval method, device, equipment and storage medium, which can combine the high efficiency of file storage and the rapidity of memory retrieval to achieve an efficient, flexible and scalable file-based fast storage and retrieval method.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a file-based fast storage and retrieval method is provided, comprising: obtaining data to be recorded; processing the data to be recorded into multiple compressed character strings with retrieval keys; creating a cache file, writing the multiple compressed character strings into the cache file, and recording the position record pointer of each compressed character string; and associating the retrieval key of each compressed character string and its position record pointer and storing them in a memory.

[0008] In one embodiment of the present disclosure, the file-based fast storage and retrieval method also includes: receiving a data read request for target data in the data to be recorded; wherein the data read request includes a target retrieval key for the target data; searching the memory for a target position record pointer corresponding to the target retrieval key; reading a target compressed string from the cache file according to the target position record pointer; and determining the target data according to the target compressed string.

[0009] In one embodiment of the present disclosure, the data to be recorded is processed into multiple compressed character strings with retrieval keys, including: serializing the data to be recorded to obtain multiple serialized data with retrieval keys; compressing the multiple serialized data to obtain multiple compressed character strings with retrieval keys.

[0010] In one embodiment of the present disclosure, the data to be recorded includes multiple objects to be recorded, and the objects to be recorded have a key-value pair data structure or a list data structure; wherein, the data to be recorded are serialized to obtain multiple serialized data with retrieval keys, including: serializing each object to be recorded to obtain serialized data of each object to be recorded; determining the retrieval key of each object to be recorded according to its key information or list index information.

[0011] In one embodiment of the present disclosure, the data to be recorded is routing concatenation data having a key-value pair data structure, and the key information of the routing concatenation data is port information used for routing concatenation.

[0012] In one embodiment of the present disclosure, the file-based fast storage and retrieval method also includes: determining the compressed character length of each serialized data after compression processing; wherein, recording the position record pointer of each compressed string includes: for each compressed string, determining the current file offset position of the cache file before the compressed string is written; determining the position record pointer of each compressed string according to the current file offset position corresponding to each compressed string and its compressed character length.

[0013] In one embodiment of the present disclosure, the file-based fast storage and retrieval method further includes: in response to completion of reading all compressed character strings in the cache file, deleting the cache file.

[0014] According to another aspect of the present disclosure, a file-based fast storage and retrieval device is provided, including: an acquisition unit for acquiring data to be recorded; a compression unit for processing the data to be recorded into multiple compressed character strings with retrieval keys; a file writing unit for creating a cache file, writing the multiple compressed character strings into the cache file, and recording the position record pointer of each compressed character string; a pointer recording unit for storing the retrieval key of each compressed character string and its position record pointer in association with each other in a memory.

[0015] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned file-based fast storage and retrieval method is implemented.

[0016] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the above-mentioned file-based fast storage retrieval method by executing the executable instructions.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the above-mentioned file-based fast storage and retrieval method.

[0018] The file-based fast storage and retrieval method provided by the embodiment of the present disclosure can first compress the data to be recorded into a small-volume compressed string, configure a retrieval key for the compressed string, and then write the compressed string into the created cache file, store the data to be recorded in the form of a cache file, and associate the retrieval key of each compressed string and its position record pointer in the cache file in the memory. It can be seen that through this method, the efficiency of file storage and the speed of memory retrieval can be combined. On the one hand, by compressing data, storage requirements can be reduced and storage efficiency can be improved. On the other hand, by establishing an index in the memory, the data retrieval speed can be increased. In addition, using the cache file as an intermediate storage medium, a large amount of data can be processed without affecting the system performance, realizing an efficient, flexible and scalable file-based fast storage and retrieval method to meet the scenario of efficient storage and fast access to large amounts of data.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0021] Figure 1 The diagram shows an exemplary system architecture to which the file-based fast storage retrieval method according to an embodiment of the present disclosure can be applied.

[0022] Figure 2 A flowchart of a file-based fast storage and retrieval method according to an embodiment of the present disclosure is shown.

[0023] Figure 3 A flowchart of reading data in a file-based fast storage and retrieval method according to an embodiment of the present disclosure is shown.

[0024] Figure 4 A flowchart of a file-based fast storage and retrieval method according to an embodiment of the present disclosure is shown.

[0025] Figure 5 A flowchart of a file-based fast storage and retrieval method according to an embodiment of the present disclosure is shown.

[0026] Figure 6 A block diagram of a file-based fast storage and retrieval device according to an embodiment of the present disclosure is shown.

[0027] Figure 7 A structural block diagram of a file-based fast storage and retrieval computer device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0029] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0030] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present disclosure, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0031] Figure 1 The diagram shows an exemplary system architecture to which the file-based fast storage retrieval method according to an embodiment of the present disclosure can be applied.

[0032] like Figure 1As shown, the system architecture may include a server 101, a network 102, and a client 103. The network 102 is used to provide a medium for a communication link between the client 103 and the server 101. The network 102 may include various connection types, such as wired, wireless communication links, or optical fiber cables.

[0033] In an exemplary embodiment, the client 103 for data transmission with the server 101 may include, but is not limited to, electronic devices of types such as smart phones, desktop computers, tablet computers, laptop computers, smart speakers, digital assistants, AR (Augmented Reality) devices, VR (Virtual Reality) devices, and smart wearable devices. Optionally, the operating system running on the electronic device may include, but is not limited to, Android system, IOS system, Linux system, Windows system, and the like.

[0034] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some practical applications, server 101 can also be a server of a network platform, which can be, for example, a trading platform, a live broadcast platform, a social platform, or a music platform, etc., which is not limited in the embodiments of the present disclosure. Among them, the server can be a single server or a cluster formed by multiple servers, and the present disclosure does not limit the specific architecture of the server.

[0035] In an exemplary embodiment, the server 101 may store data to be recorded, and the user may initiate a data recording request through the client 103, so as to utilize the file-based fast storage and retrieval method provided by the present invention to first quickly store the data to be recorded in the form of a cache file to the local memory; and then initiate a data reading request through the client 103 to read the required data according to the cache file in the local memory.

[0036] In an exemplary embodiment, the process of the client 103 for implementing a file-based fast storage and retrieval method may be: the client 103 obtains the data to be recorded; the client 103 processes the data to be recorded into multiple compressed strings with retrieval keys; the client 103 creates a cache file, writes the multiple compressed strings into the cache file, and records the location record pointer of each compressed string; the client 103 associates and stores the retrieval key of each compressed string and its location record pointer in the memory.

[0037] In addition, it should be noted that Figure 1 What is shown is only one application environment of the file-based fast storage and retrieval method provided by the present disclosure. Figure 1 The numbers of servers 101, networks 102 and clients 103 are merely illustrative, and any number of clients, networks and servers may be provided according to actual needs.

[0038] In order to enable ordinary persons in the art to better understand the technical solution of the present disclosure, the various steps of the file-based fast storage and retrieval method in the exemplary embodiment of the present disclosure will be described in more detail below with reference to the accompanying drawings and embodiments.

[0039] Figure 2 A flowchart of a file-based fast storage and retrieval method according to an embodiment of the present disclosure is shown. The method provided by the embodiment of the present disclosure can be performed as follows: Figure 1 The server 101 or the client 103 shown is executed, but the present disclosure is not limited to this.

[0040] In the following examples, the server 101 is used as the execution subject for illustration.

[0041] like Figure 2 As shown, the file-based fast storage and retrieval method provided by the embodiment of the present disclosure may include the following steps.

[0042] Step S201, obtaining data to be recorded.

[0043] In the disclosed embodiment, data to be recorded may be collected or generated first, and the data to be recorded is data that needs to be recorded, synchronized or stored. The data may come from various sources, such as business system import, user input, sensor readings, database query results, etc.

[0044] Step S203: Process the data to be recorded into a plurality of compressed character strings having search keys.

[0045] In the disclosed embodiment, the acquired data to be recorded can be segmented according to certain rules and converted into compressed strings. Each compressed string can correspond to a search key (or index key), which can be a unique identifier generated according to the string content or related attributes, or an identifier assigned by the system in a preset manner.

[0046] In an exemplary embodiment, the data to be recorded may include multiple record objects, and the data may be directly processed based on the record object as the granularity, and each record object may be processed into a compressed string, wherein the storage space occupied by the compressed string is much smaller than that of the original record object.

[0047] In this embodiment, the original data to be recorded is processed and compressed into a string format, which not only helps to save storage space, but also reduces the bandwidth requirement during data transmission. Each compressed string has a search key, which can ensure that each compressed string can be uniquely and quickly identified so as to be quickly retrieved in subsequent steps.

[0048] Step S205: create a cache file, write the multiple compressed character strings into the cache file, and record the position record pointer of each compressed character string.

[0049] In the disclosed embodiment, one or more cache files may be created on a disk to store compressed data strings until the compressed strings or the data to be recorded are permanently saved or further processed. When writing compressed strings to a file, the offset position of each compressed string in the file may be determined and then recorded as a position record pointer. When subsequently retrieving data, these pointers may be used to quickly locate the position of the data in the file.

[0050] Through this embodiment, the data to be recorded can be recorded in the form of a cache file. Since the data volume of the compressed string is much smaller than the original data before compression, the storage space occupied by the cache file will also be much smaller than the original data to be recorded.

[0051] Step S207: associate and store the search key of each compressed character string and its position record pointer in the memory.

[0052] In the disclosed embodiment, a data structure (such as a hash table, a dictionary, etc.) can be established in a memory (such as a designated resource pool) to associate and store a search key with a location record pointer of a corresponding compressed string in a cache file. In this way, when it is necessary to search for data according to the search key, the location record pointer can be directly obtained from the memory, and then the corresponding data in the cache file can be quickly located.

[0053] In an exemplary embodiment, the method can be applied to scenarios that require rapid storage and retrieval of large amounts of data, or scenarios that require frequent read and write operations and have high requirements for response time, such as real-time data analysis, log management, database indexing, full-scale routing concatenation, and other scenarios.

[0054] Through the file-based fast storage and retrieval method provided by the present disclosure, the data to be recorded can be first compressed into a small-volume compressed string, and a retrieval key can be configured for the compressed string. Then, the compressed string can be written into the created cache file, and the data to be recorded can be stored in the form of a cache file, and the retrieval key of each compressed string and its position record pointer in the cache file can be associated and stored in the memory. It can be seen that through this method, the efficiency of file storage and the speed of memory retrieval can be combined. On the one hand, by compressing data, storage requirements can be reduced and storage efficiency can be improved. On the other hand, by establishing an index in the memory, the data retrieval speed can be increased. In addition, using the cache file as an intermediate storage medium, a large amount of data can be processed without affecting the system performance, realizing an efficient, flexible and scalable file-based fast storage and retrieval method to meet the scenario of efficient storage and fast access to large amounts of data.

[0055] Figure 3 A flowchart of reading data in a file-based fast storage and retrieval method according to an embodiment of the present disclosure is shown.

[0056] In some embodiments, the file-based fast storage retrieval method further includes the following steps.

[0057] Step S301, receiving a data reading request for target data in the data to be recorded; wherein the data reading request includes a target search key of the target data.

[0058] In the disclosed embodiment, a data read request from a user or other service may be received, and the request may include a target search key of the target data to be retrieved. A correct target search key can ensure that the target data can be located in the cache data.

[0059] Step S303, searching the memory for a target position record pointer corresponding to the target search key.

[0060] In the disclosed embodiment, the associated data structure (such as a hash table, a dictionary, etc.) previously stored in the memory can be used to search according to the target search key in the request to obtain the location record pointer of the target data in the cache file. Since the memory has a high-speed access characteristic, the access speed is very fast, so the target location record pointer can be quickly found in this step.

[0061] In addition, you need to ensure that the search key matches accurately to find the correct data location.

[0062] In an exemplary embodiment, if the target search key carried in the data read request is incorrect, the corresponding target location record pointer may not be found correctly, and the search result may be returned in a preset manner, such as returning a null value. For example, if the data provided by the network manager when issuing the request is different from the data stored in the actual resource pool, the object ID (i.e., the target search key) provided by the former may not be found in the latter.

[0063] Step S305 reads the target compressed string from the cache file according to the target position record pointer.

[0064] In the disclosed embodiment, once the target location record pointer is found, the corresponding compressed string can be accurately located and read directly from the cache file according to the location indicated by the pointer. This step can utilize the offset position information of the previously stored data in the cache file to avoid scanning the file in full, thereby greatly improving the data retrieval efficiency.

[0065] In an exemplary embodiment, the cache file can be accessed through a file I / O operation using the found position record pointer, the cache file can be opened and located to the starting position of the target data, and then the compressed string of the target data can be read from the starting position.

[0066] Step S307: determining the target data according to the target compressed character string.

[0067] In the disclosed embodiment, the compressed string read can be decompressed to restore the original target data. After the target data is restored, the data can be returned to the requester, or subsequent data synchronization or storage processing can be performed in response to other instructions, such as using the target data for other services (such as routing concatenation), or storing the target data in other databases of the current device.

[0068] If the data is encrypted or converted in other forms before compression, this step can use the same algorithm or inverse algorithm as that used for compression to decompress the string to ensure that the decompressed data is consistent with the original data. For example, if serialization and compression are used in the process of "processing the data to be recorded into multiple compressed strings with search keys", the same or corresponding algorithm can be used in this embodiment to perform decompression and deserialization to restore the data.

[0069] Through this embodiment, the file-based fast storage and retrieval method provided by the present disclosure can efficiently retrieve and return corresponding data according to the search key provided by the user, which not only realizes efficient storage of data, but also realizes fast retrieval and recovery of data. This capability is crucial for applications that need to frequently access large amounts of data, such as real-time monitoring systems, data analysis platforms, etc. Through the combination of memory-accelerated retrieval and file persistent storage, this method can efficiently retrieve data while ensuring data reliability, providing an efficient and reliable data storage and retrieval solution.

[0070] In some embodiments, the file-based fast storage and retrieval method further includes: in response to completion of reading all compressed character strings in the cache file, deleting the cache file.

[0071] In the disclosed embodiment, deleting cache files that have been read can help free up storage space and ensure that the storage space is effectively used. And by automatically deleting the read files, the burden of manual file management can be reduced, making the data management process more concise and efficient. In addition, if the cache files contain sensitive information, deleting these cache files in a timely manner can reduce the risk of data leakage and enhance the security of the system.

[0072] In an exemplary embodiment, a specific field may also be configured to record which cache files have been completely read and which files still contain unprocessed data, so as to ensure through this field that all compressed strings have been correctly read and processed before deleting the cache files.

[0073] Figure 4 A flowchart of a file-based fast storage and retrieval method according to an embodiment of the present disclosure is shown.

[0074] In the disclosed embodiment, Figure 4 Steps S401, S407, and S409 in the file-based rapid storage and retrieval method are respectively Figure 2 Steps S201, S205, and S207 in the file-based rapid storage and retrieval method are corresponding and will not be repeated here.

[0075] In the embodiment of the present disclosure, Figure 2 Based on the fast file-based storage and retrieval method shown, Figure 4 The file-based fast storage retrieval method may further include the following steps.

[0076] Step S403: serialize the data to be recorded to obtain a plurality of serialized data with search keys.

[0077] In the disclosed embodiment, the original data to be recorded may be serialized first. Serialization is the process of converting a data structure or object state into a format that can be stored or transmitted (such as JSON, XML, binary, etc.). In this step, the converted serialized data may be a linear format that is easy to store and transmit, such as a byte stream or a string. Each serialized data may be associated with a retrieval key, which is a unique identifier of the data, so that the data can be retrieved based on this key later.

[0078] In an exemplary embodiment, Alibaba Fastjson (a high-performance, fully functional JSON library written in Java) may be used for serialization processing to convert Java objects into strings in JSON format for storage or network transmission.

[0079] In this embodiment, serialization processing not only enables data to be stored in a unified manner, but also ensures the integrity and consistency of the data, facilitating subsequent compression and processing.

[0080] Step S405: compress the plurality of serialized data to obtain a plurality of compressed character strings having search keys.

[0081] In the disclosed embodiment, the serialized data can be compressed to reduce the data size, save storage space and improve transmission efficiency. The serialized data usually has a certain degree of redundancy, so its volume can be further reduced by a compression algorithm. In addition, the compressed data block (i.e., the compressed string) can still retain its original search key to maintain the searchability of the data.

[0082] In an exemplary embodiment, a compression algorithm suitable for data characteristics and compression requirements can be selected to implement compression processing. For example, a lossless compression algorithm (such as GZIP, DEFLATE, etc.) can be used to ensure that the data can be completely restored to its original state after decompression.

[0083] Through this embodiment, an effective data preprocessing mechanism is provided. Among them, through serialization, various types of data can be processed and converted into a unified format; through compression, the use of storage space can be optimized and the efficiency of data transmission can be improved, which can meet the application scenarios that need to store and transmit large amounts of data.

[0084] In some embodiments, the data to be recorded includes multiple objects to be recorded, and the objects to be recorded have a key-value pair data structure or a list data structure; wherein, the data to be recorded are serialized to obtain multiple serialized data with retrieval keys, including: serializing each object to be recorded to obtain serialized data of each object to be recorded; determining the retrieval key according to the key information or list index information of each object to be recorded.

[0085] In an embodiment of the present disclosure, the data to be recorded may include multiple objects to be recorded having a key-value pair data structure; wherein, serializing the data to be recorded to obtain multiple serialized data with retrieval keys may include: serializing each object to be recorded to obtain the serialized data of each object to be recorded, and determining the retrieval key according to the key in the key-value pair of each object to be recorded.

[0086] Among them, the object with a key-value pair data structure can be, for example, a dictionary, a hash table, and other data. For this type of data to be recorded, a key (or a combination of multiple keys) in the object can be directly used as a search key. In addition, it should be ensured that the key is unique, or its combination is unique in the data set, so that the corresponding object can be accurately retrieved.

[0087] In some embodiments, the data to be recorded is routing concatenation data having a key-value pair data structure, and the key information of the routing concatenation data is port information used for routing concatenation.

[0088] Among them, end-to-end management, as an important technology for OTN (Optical Transport Network) intelligent operation, can provide users with a hierarchical end-to-end service management view, which is of great significance for improving operation and maintenance efficiency. The core technology of end-to-end management is routing concatenation. For a complex OTN service, a large amount of basic data needs to be cached during the routing concatenation process, such as topological connection, cross-connection and protection group data.

[0089] In the disclosed embodiment, the routing concatenation data may be these basic data, which may describe the path of a data packet from a source to a destination in a network, and may include a series of network nodes (such as routers, switches, etc.) and connection information between them. For the routing concatenation data, the key information may be information representing a network port, and the value information may be routing information associated with the port or the address of the next hop.

[0090] Through this embodiment, in the routing cascade of the full SNC of the ultra-large-scale OTN network, this routing cascade data processing method using port information as the key can solve the memory expansion problem, and can significantly improve the storage and retrieval efficiency of network routing information, maintaining a high routing cascade efficiency.

[0091] In an embodiment of the present disclosure, the data to be recorded may include multiple objects to be recorded having a list data structure; wherein, serializing the data to be recorded to obtain multiple serialized data with retrieval keys may include: serializing each object to be recorded to obtain the serialized data of each object to be recorded, and determining the retrieval key of each object to be recorded according to the list index value of each object to be recorded.

[0092] Among them, the object with list data structure can be, for example, array, list and other data. For this type of data to be recorded, since the elements in the list are usually accessed by index, the index information of the list (or some combination of the index and the element content) can be used as the search key. In addition, if the list is mutable, the index may change (for example, through insertion or deletion operations). In this case, it can also be considered to assign a unique identifier to each element in the list as a search key during the serialization process.

[0093] Through the disclosed embodiment, no matter whether the object to be recorded is a key-value pair or a list data structure, it can be effectively converted into a format suitable for storage and retrieval. This embodiment not only improves the efficiency of data storage, but also ensures the accuracy and speed of data retrieval, and can be used as an effective strategy for processing complex data structures.

[0094] In some embodiments, the file-based fast storage and retrieval method also includes: determining the compressed character length of each serialized data after compression processing; wherein, recording the position record pointer of each compressed string includes: for each compressed string, determining the current file offset position of the cache file before the compressed string is written; determining the position record pointer of each compressed string according to the current file offset position corresponding to each compressed string and its compressed character length.

[0095] In the disclosed embodiment, after the serialized data is compressed, the character length of each compressed string can be determined. This length information can be used to accurately locate and read the compressed string in the cache file. In addition, before the compressed string is written to the cache file, the current offset position of the file (i.e., the position of the file pointer) can be recorded, and this offset position represents the starting point of the compressed string to be written in the file. Combining the file offset position before each compressed string is written and the compressed character length of the string, a position record pointer can be accurately determined.

[0096] In the disclosed embodiment, the position record pointer may indicate a range or interval, which includes the offset position of the cache file and the length of the string, so as to indicate the exact position of each compressed string in the cache file. Through the position record pointer, it is possible to directly jump to the correct position in the file and only read the data of the required length instead of the entire file, so as to retrieve the required target data more efficiently and accurately.

[0097] In addition, this embodiment also provides better data integrity and consistency assurance. In this embodiment, it can be verified whether the read data length matches the expected compressed character length, thereby detecting any potential data corruption or loss.

[0098] In an exemplary embodiment, in order to process the data to be recorded in the key-value pair data structure, a class FileMap may be defined based on the Map interface for fast access to data. Specifically, the following content may be defined:

[0099] ①Define the Map interface implementation class FileMap, and inherit ConcurrentHashMap to implement the Map interface.

[0100] ②Define the internal hash table variable seekMap to save and read record pointers.

[0101] Based on this, the methods supported by the FileMap class based on the Map interface implemented in this embodiment may include:

[0102] ①put(K key,V value): write the record value to the file and update the relationship between the current record key and record pointer of seekMap.

[0103] ②get(K key): Get the record pointer from seekMap according to key. If there is one, parse the record from the file and return it.

[0104] ③clear(): delete the current cache file and clear the hash table seekMap.

[0105] Figure 5 A flowchart of a file-based fast storage and retrieval method according to an embodiment of the present disclosure is shown.

[0106] In an exemplary embodiment, the file-based fast storage and retrieval method provided by the present disclosure can be applied to the scenario of full-volume routing concatenation, such as Figure 5 As shown, in this scenario, the file-based fast storage and retrieval method may include the following steps.

[0107] Step S501, initializing basic data to file cache.

[0108] Among them, the basic data for routing concatenation can be extracted from the source database, and after the basic data is serialized and compressed, it is saved in the file cache that implements the FileMap interface.

[0109] Step S503, perform concatenation according to the subnet connection, and obtain data required for concatenation from the file cache. After the concatenation is completed, clear the temporary cache of the current subnet connection.

[0110] Before connecting the subnets in series, you can first read the detailed information and bearer relationship of the current subnet connection from the file cache. During the connection process, according to the next-hop port information, retrieve and read the required basic data from the file cache of the Map interface. The basic data includes but is not limited to key information such as topology structure, protection group configuration, LAG member objects, binding relationship, and service layer configuration. After the connection work is completed, the temporary cache of the current subnet connection can be cleared in time to ensure efficient use of resources.

[0111] Step S505, after the full concatenation is completed, clear the file and memory cache.

[0112] Among them, after the full concatenation is completed, the cached file can be deleted and the record pointer object in the memory can be cleared (that is, the associated stored search key and the corresponding location record pointer are deleted).

[0113] In an exemplary embodiment, in order to process the data to be recorded in the list data structure, a class FileList can be defined based on the List interface for fast access to data. Specifically, the following content can be defined:

[0114] ①Define the List interface implementation class FileList, which inherits Vector to implement the List interface.

[0115] ②Define the internal array variable seekList to save and read record pointers.

[0116] Based on this, the methods supported by the FileList class based on the List interface implemented in this embodiment may include:

[0117] ①add(T obj): Every time a record is added, a record pointer will be added to seekList to record the location information of each record in the file.

[0118] ②get(int index): According to the input index, if the parameter is within the size range of the seekList array, get the record pointer from the seekList, and then get the record from the file and return it.

[0119] ③clear(): delete the current cache file and clear the array seekList.

[0120] In an exemplary embodiment, the file-based fast storage retrieval method provided by the present disclosure can be applied to the scenario of full synchronization of primary and standby databases. In this scenario, the file-based fast storage retrieval method can include the following steps.

[0121] 1. For each table to be synchronized, you can create a file cache that implements the FileList interface in advance to temporarily store data read from the main database.

[0122] 2. Use the cursor mechanism to query data from the master database one by one. In order to improve synchronization efficiency, batch processing can be used. For example, after reading a specified number of data (such as 500 or 1,000), the data can be encapsulated into a List object and saved to the previously created file cache. At the same time, a task to write to the temporary table of the slave database can be submitted asynchronously, where the input parameter is the index position of the current List object in the file cache.

[0123] 3. The task of writing to the temporary table of the slave library can be executed according to the passed-in index, that is, the corresponding List object can be queried from the file cache. Then, the task will parse the record objects in the List and write them in batches to the temporary table of the slave library.

[0124] In this step, the efficiency of the file cache is utilized to avoid waiting for a long time for querying the main database, and to avoid synchronous processing of querying from the main database and writing from the slave database, and asynchronous processing is implemented, thereby improving the data synchronization speed.

[0125] 4. The slave database can compare the data of the temporary table with the formal table. Based on the comparison results, the slave database can confirm that the synchronization of the table data is complete and ensure the data consistency between the master and slave databases. After that, the file cache that implements the FileList interface created in step 1 can be deleted.

[0126] Through this embodiment, the data synchronization efficiency can be significantly improved, and the data transmission and processing overhead can be reduced. At the same time, the use of file cache can also provide greater flexibility and scalability, making the synchronization process more reliable and stable.

[0127] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0128] Figure 6FIG. 6 is a block diagram of a file-based fast storage and retrieval device 600 according to an embodiment of the present disclosure; Figure 6 As shown, it includes: an acquisition unit 601, used for acquiring data to be recorded; a compression unit 602, used for processing the data to be recorded into multiple compressed character strings with retrieval keys; a file writing unit 603, used for creating a cache file, writing the multiple compressed character strings into the cache file, and recording the position record pointer of each compressed character string; a pointer recording unit 604, used for storing the retrieval key of each compressed character string and its position record pointer in association with each other in the memory.

[0129] Through the file-based fast storage and retrieval device provided by the present disclosure, the data to be recorded can be compressed into a small-volume compressed string, and a retrieval key can be configured for the compressed string. Then, the compressed string can be written into the created cache file, and the data to be recorded can be stored in the form of a cache file, and the retrieval key of each compressed string and its position record pointer in the cache file can be associated and stored in the memory. It can be seen that through this method, the efficiency of file storage and the speed of memory retrieval can be combined. On the one hand, by compressing data, storage requirements can be reduced and storage efficiency can be improved. On the other hand, by establishing an index in the memory, the data retrieval speed can be increased. In addition, using a cache file as an intermediate storage medium, a large amount of data can be processed without affecting system performance, realizing an efficient, flexible and scalable file-based fast storage and retrieval method to meet the scenario of efficient storage and fast access to large amounts of data.

[0130] In some embodiments, the file-based fast storage and retrieval device 600 also includes: a search unit 605, a reading unit 606, and a decompression unit 607; the acquisition unit 601 is also used to: receive a data read request for target data in the data to be recorded; wherein the data read request includes a target retrieval key of the target data; the search unit 605 is used to search the target position record pointer corresponding to the target retrieval key in the memory; the reading unit 606 is used to read the target compressed string from the cache file according to the target position record pointer; the decompression unit 607 is used to determine the target data based on the target compressed string.

[0131] In some embodiments, the compression unit 602 processes the data to be recorded into multiple compressed character strings with retrieval keys, including: serializing the data to be recorded to obtain multiple serialized data with retrieval keys; compressing the multiple serialized data to obtain multiple compressed character strings with retrieval keys.

[0132] In some embodiments, the data to be recorded includes multiple objects to be recorded, and the objects to be recorded have a key-value pair data structure or a list data structure; wherein the compression unit 602 serializes the data to be recorded to obtain multiple serialized data with retrieval keys, including: serializing each object to be recorded to obtain serialized data of each object to be recorded; determining the retrieval key according to the key information or list index information of each object to be recorded.

[0133] In some embodiments, the data to be recorded is routing concatenation data having a key-value pair data structure, and the key information of the routing concatenation data is port information used for routing concatenation.

[0134] In some embodiments, the compression unit 602 is also used to: determine the compressed character length of each serialized data after compression processing; wherein the pointer recording unit 604 records the position record pointer of each compressed string, including: for each compressed string, determining the current file offset position of the cache file before the compressed string is written; determining the position record pointer of each compressed string according to the current file offset position corresponding to each compressed string and its compressed character length.

[0135] In some embodiments, the file-based fast storage and retrieval device 600 further includes: a deleting unit 608, which is used to delete the cache file in response to completion of reading all compressed character strings in the cache file.

[0136] Figure 6 For other contents of the embodiment, reference can be made to the other embodiments mentioned above.

[0137] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0138] Figure 7 The structural block diagram of a computer device for fast storage and retrieval based on files in the embodiment of the present disclosure is shown. It should be noted that the electronic device shown in the figure is only an example and should not bring any limitation to the functions and scope of use of the embodiment of the present invention.

[0139] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present invention will be described. Figure 7 The electronic device 700 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0140] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710).

[0141] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 710 can perform the following steps: Figure 2 The method shown in .

[0142] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .

[0143] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0144] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0145] The electronic device 700 may also communicate with one or more external devices 800 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0146] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0147] The program product for implementing the above method according to the embodiment of the present invention may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0148] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0149] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0150] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0151] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0152] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0153] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0154] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0155] According to one aspect of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional implementations of the above-mentioned embodiments.

[0156] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A file-based rapid storage and retrieval method, characterized in that: include: Get the data to be recorded; Processing the data to be recorded into a plurality of compressed character strings with search keys; Creating a cache file, writing the plurality of compressed character strings into the cache file, and recording a position record pointer of each compressed character string; The search key of each compressed string and its position record pointer are associated and stored in the memory.

2. The method according to claim 1, characterized in that The method further comprises: receiving a data reading request for target data in the data to be recorded; wherein the data reading request includes a target search key of the target data; searching the memory for a target location record pointer corresponding to the target search key; Reading a target compressed string from the cache file according to the target position record pointer; The target data is determined according to the target compressed character string.

3. The method according to claim 1 or 2, characterized in that: Processing the data to be recorded into a plurality of compressed character strings with search keys, including: Serializing the data to be recorded to obtain a plurality of serialized data with retrieval keys; Compression processing is performed on multiple serialized data to obtain multiple compressed character strings with search keys.

4. The method according to claim 3, characterized in that The data to be recorded includes a plurality of objects to be recorded, and the objects to be recorded have a key-value pair data structure or a list data structure; The data to be recorded is serialized to obtain a plurality of serialized data with search keys, including: Serializing each object to be recorded to obtain serialized data of each object to be recorded; The search key is determined according to the key information or list index information of each object to be recorded.

5. The method according to claim 4, characterized in that The data to be recorded is routing serial connection data having a key-value pair data structure, and the key information of the routing serial connection data is the port information used for routing serial connection.

6. The method according to claim 3, characterized in that The method further comprises: determining the compressed character length of each serialized data after the compression process; Among them, the location record pointer of each compressed string includes: For each compressed string, determining a current file offset position of the cache file before the compressed string is written; The position record pointer of each compressed string is determined according to the current file offset position corresponding to each compressed string and the compressed character length thereof.

7. The method according to claim 2, characterized in that The method further comprises: In response to completion of reading all compressed character strings in the cache file, the cache file is deleted.

8. A file-based fast storage and retrieval device, characterized in that: include: An acquisition unit, used for acquiring data to be recorded; A compression unit, used for processing the data to be recorded into a plurality of compressed character strings with search keys; A file writing unit, used for creating a cache file, writing the plurality of compressed character strings into the cache file, and recording a position record pointer of each compressed character string; The pointer recording unit is used to associate and store the search key of each compressed string and its position record pointer in the memory.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the file-based fast storage and retrieval method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the file-based fast storage and retrieval method as described in any one of claims 1 to 7.

Images