Data processing method, device, equipment and computer readable storage medium

Abstract

The present invention relates to the field of financial technology, and discloses a data processing method, device, equipment and computer-readable storage medium. The data processing method comprises: obtaining data to be stored, storing the data to be stored in a preset key-value memory in the form of a key-value pair, and obtaining a corresponding first storage address; parsing the data to be stored to obtain a first keyword; establishing an index between the first keyword and the first storage address, and storing it in a preset keyword indexer. The present invention can solve the problem of poor flexibility of the existing memory structure data storage method.

Description

Technical Field

[0001] The present invention relates to the technical field of financial technology (Fintech), and in particular to a data processing method, device, equipment and computer-readable storage medium. Background Art

[0002] With the development of computer technology, more and more technologies are being applied in the financial field. The traditional financial industry is gradually transforming into financial technology (Fintech). However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on technology.

[0003] The more common existing memory structures include Map (associative container), array, binary tree, etc., which can support searching by key, by index, and by node value respectively. However, the current memory structure can only support one way of searching. For example, Map supports finding the corresponding value by key, and array supports quickly finding the corresponding value by index. Afterwards, when searching for data by value, it is necessary to traverse the entire memory structure, which has low query efficiency, poor performance and waste of CPU resources. Therefore, the existing memory structure has poor flexibility in data storage. Summary of the invention

[0004] The main purpose of the present invention is to provide a data processing method, device, equipment and computer-readable storage medium, aiming to solve the problem of poor flexibility of existing memory structure data storage methods.

[0005] To achieve the above object, the present invention provides a data processing method, the data processing method comprising:

[0006] Acquire the data to be stored, store the data to be stored in a preset key-value memory in the form of a key-value pair, and acquire the corresponding first storage address;

[0007] Parsing the data to be stored to obtain a first keyword;

[0008] An index is established between the first keyword and the first storage address, and is stored in a preset keyword indexer.

[0009] Optionally, the step of establishing an index between the first keyword and the first storage address and storing the index in a preset keyword indexer includes:

[0010] Determining whether the first keyword exists in the preset keyword indexer;

[0011] If it exists, then adding an index between the first storage address and the first keyword to the index corresponding to the first keyword;

[0012] If it does not exist, a new index is created between the first keyword and the first storage address and stored in the preset keyword indexer.

[0013] Optionally, the data processing method further includes:

[0014] When receiving a data reading request, obtaining data reading information carried in the data reading request;

[0015] If the data read information is the second storage address of the data to be searched, then the first target value corresponding to the second storage address is searched from the preset key-value memory and returned to the query end corresponding to the data read request;

[0016] If the data read information is a second keyword of the data to be searched, then searching the preset keyword indexer to obtain a target storage address corresponding to the second keyword;

[0017] A second target value corresponding to the target storage address is searched from the preset key-value memory and returned to the query end corresponding to the data read request.

[0018] Optionally, the data processing method further includes:

[0019] When receiving a data deletion request, obtaining a third storage address of the data to be deleted according to the data deletion request;

[0020] Searching for a third target value corresponding to the third storage address from the preset key-value pair memory and deleting the third target value;

[0021] Parsing the third target value to obtain a third keyword;

[0022] The third keyword is searched from the preset keyword indexer, and index information between the third keyword and the third storage address is deleted.

[0023] Optionally, the data processing method further includes:

[0024] When receiving a data modification request, obtaining target modification data and its fourth storage address, and modified data according to the data modification request;

[0025] The fourth storage address is used as the third storage address, and the steps are performed: searching the preset key-value pair memory for a third target value corresponding to the third storage address, and deleting the third target value;

[0026] After the deletion is completed, the modified data is used as the data to be stored, and the steps are performed: the data to be stored is stored in a preset key-value storage device in the form of a key-value pair, and the corresponding first storage address is obtained.

[0027] Optionally, the data processing method further includes:

[0028] When a data change event is detected, the memory occupancy rate and the total memory occupancy rate of the corresponding keyword are updated according to the data change event.

[0029] Optionally, the data processing method further includes:

[0030] Obtaining the current total memory usage in real time, and determining whether the current total memory usage exceeds a preset threshold;

[0031] If the current total memory usage exceeds a preset threshold, the stored data is cleaned up according to a preset cleaning rule; wherein the preset cleaning rule includes one or more of the following:

[0032] Clean up the storage data corresponding to the keyword with the largest memory usage, clean up the storage data corresponding to the keyword that has not been used within a preset time period, and clean up the storage data corresponding to the keyword with the lowest usage rate.

[0033] Optionally, before the step of storing the data to be stored in a preset key-value memory in the form of a key-value pair and obtaining the corresponding first storage address, the method further includes:

[0034] Obtaining a classification label of the data to be stored, and determining a target key-value storage device and a target keyword indexer according to the classification label;

[0035] The step of storing the data to be stored in a preset key-value memory in the form of a key-value pair and obtaining the corresponding first storage address comprises:

[0036] Storing the data to be stored in the target key-value storage device in the form of a key-value pair, and obtaining a corresponding first storage address;

[0037] The step of establishing an index between the first keyword and the first storage address and storing the index in a preset keyword indexer includes:

[0038] An index is established between the first keyword and the first storage address, and stored in the target keyword indexer.

[0039] In addition, to achieve the above object, the present invention further provides a data processing device, the data processing device comprising:

[0040] A first acquisition module is used to acquire data to be stored, store the data to be stored in a preset key-value memory in the form of a key-value pair, and acquire a corresponding first storage address;

[0041] A first parsing module, used for parsing the data to be stored to obtain a first keyword;

[0042] An index building module is used to build an index between the first keyword and the first storage address, and store it in a preset keyword indexer.

[0043] In addition, to achieve the above-mentioned purpose, the present invention also provides a data processing device, which includes: a memory, a processor, and a data processing program stored in the memory and executable on the processor, and the data processing program implements the steps of the data processing method described above when executed by the processor.

[0044] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a data processing program is stored, and when the data processing program is executed by a processor, the steps of the data processing method described above are implemented.

[0045] The present invention provides a data processing method, device, equipment and computer-readable storage medium, which obtains data to be stored, stores the data to be stored in a preset key-value memory in the form of a key-value pair, and obtains a corresponding first storage address; parses the data to be stored to obtain a first keyword; establishes an index between the first keyword and the first storage address, and stores it in a preset keyword indexer. In the present invention, while using a key-value pair to store the data to be stored, a storage address is obtained, and an index between the storage address and the keyword is established and stored. Through the above method, a new memory structure is realized, which supports storing data in a preset key-value memory according to a key to obtain a storage address, and also supports parsing data to obtain a keyword, indexing the keyword and the storage address and storing them in a preset keyword indexer. Subsequently, it supports fast search for data according to the key, and also supports fast search for data according to the keyword index. It has good flexibility and versatility and can be applied to different query scenarios. In addition, when searching for data by value in the future, the storage address of the value to be searched can be quickly queried according to the index based on the keyword of the value to be searched, and then the data to be searched can be quickly queried without traversing the entire memory structure, thereby improving the search efficiency and avoiding the problems of low performance and waste of CPU resources due to traversal query in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention;

[0047] Figure 2 It is a flowchart of a first embodiment of a data processing method of the present invention;

[0048] Figure 3 A schematic diagram of a data processing memory structure involved in the data processing method of the present invention;

[0049] Figure 4 FIG. 1 is a schematic diagram of functional modules of a first embodiment of a data processing device according to the present invention.

[0050] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0052] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0053] The data processing device in the embodiment of the present invention may be a smart phone, or may be a terminal device such as a PC (Personal Computer), a tablet computer, or a portable computer.

[0054] like Figure 1 As shown, the data processing device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0055] Those skilled in the art will understand that Figure 1 The data processing device structure shown in the figure does not constitute a limitation of the data processing device, and may include more or less components than shown in the figure, or combine some components, or arrange the components differently.

[0056] like Figure 1As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, and a data processing program.

[0057] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; and the processor 1001 can be used to call the data processing program stored in the memory 1005 and execute the various steps of the following data processing method.

[0058] Based on the above hardware structure, various embodiments of the data processing method of the present invention are proposed.

[0059] The invention provides a data processing method.

[0060] Reference Figure 2 , Figure 2 FIG. 1 is a flow chart of a first embodiment of a data processing method according to the present invention.

[0061] In this embodiment, the data processing method includes:

[0062] Step S10, obtaining data to be stored, storing the data to be stored in a preset key-value memory in the form of a key-value pair, and obtaining a corresponding first storage address;

[0063] The data processing method of this embodiment is implemented by a data processing device, and the memory structure of the device is set as follows Figure 3The data processing memory structure shown is divided into three layers. The first layer is a keyword parser (Keyword Parser) and a key value finder (Key Finder), wherein the keyword parser is used to parse the data to be stored to obtain keywords, and the key value finder is used to find the corresponding value (the value in this embodiment is the storage data) through the key (key, the key in this embodiment is the storage address), or first obtain the corresponding storage address from the keyword indexer set through the keyword, and then obtain the corresponding value from the key-value pair memory set; the second layer is a key-value memory set (Key-Value Map Set) and a keyword indexer set (Inverted Index Set), the key-value memory set may include multiple categories of key-value memories, and similarly, the keyword indexer set may include multiple categories of keyword indexers to achieve classified storage. Of course, in specific implementation, it may also include only one key-value memory and one keyword indexer; the third layer is a specific type of key-value memory (Key-Value Map) and keyword indexer (Inverted Index Set). Index), wherein the key-value memory is used to store the data content as key-value pairs, and the keyword indexer is used to store the mapping relationship between the keyword and the data content storage address. In addition, the data processing memory structure also includes a listener (Listener) and an automatic cleaner (AutoClear), wherein the listener is used to update the memory occupancy rate and the total memory occupancy rate of the corresponding keyword after the operations such as data storage, data deletion and data modification are completed; and is also used to update the usage information of the corresponding keyword (including but not limited to the number of uses, usage rate and usage time) after the data reading operation is completed. The automatic cleaner is used to clean and recycle the data when the total memory occupancy rate is monitored to exceed the preset threshold, so as to ensure the stable and normal operation of the program.

[0064] It should be noted that the present invention is applied in a memory structure rather than a storage structure such as a disk or memory. Conventionally, a disk or memory stores data in sequence, and the internal partitions are also called in sequence, that is, when calling data, only the data of adjacent partitions can be called. The present invention is applied in a memory structure, and a storage method of the memory structure is designed. The calls between data are not limited to the sequential calling method, but can be called and obtained across zones, thereby improving the efficiency of data calling.

[0065] In this embodiment, when storing data, the data to be stored is first obtained, and then the data to be stored is stored in a preset key-value memory in the form of a key-value pair, and the corresponding storage address is obtained (to distinguish it from the storage address involved later, it is recorded as the first storage address). Specifically, the first storage address is used as the key and the data to be stored is used as the value to form a key-value pair, and storage is performed in this form.

[0066] Step S20, parsing the data to be stored to obtain a first keyword;

[0067] Then, the data to be stored is parsed to obtain a corresponding keyword, which is recorded as the first keyword (to distinguish it from the keywords involved subsequently).

[0068] Specifically, during parsing, the data to be stored can be directly segmented by a preset segmentation algorithm or segmentation tool to obtain keywords. The preset segmentation algorithm may include but is not limited to: a maximum matching segmentation algorithm, a segmentation algorithm based on HMM (Hidden Markov Model), a segmentation algorithm based on CRF (Conditional Random Fields) model, and a custom segmentation algorithm (such as by specifying annotations for generating keywords when defining objects to be stored, such as specifying delimiters, regular expression values, etc.), etc. The preset segmentation tools include but are not limited to: IK (a Chinese segmenter based on dictionaries and rules) segmenter, Chinese lexical analysis system ICTCLAS, Tsinghua University Chinese lexical analysis program THULAC, language technology platform LTP, etc.

[0069] In addition, when parsing, after word segmentation, the obtained word segmentation can be screened to obtain keywords. The screening method includes but is not limited to: removing stop words, and screening out words with TF-IDF (term frequency-inverse document frequency) values ​​less than a preset threshold. By screening, some meaningless and unimportant words can be prevented from occupying memory.

[0070] Step S30: establishing an index between the first keyword and the first storage address, and storing the index in a preset keyword indexer.

[0071] Finally, an index is created between the first keyword and the first storage address, and stored in a preset keyword indexer. Specifically, the index is created with the first keyword as the index value and the first storage address as the value, such as Figure 3 For example, the corresponding first keyword is obtained by parsing the data to be stored (recorded as data), and the address of the data to be stored is recorded as address, and a key-value pair address-data can be generated, and an index keyword-address can be generated.

[0072] Specifically, step S30 includes:

[0073] Step a41, determining whether the keyword exists in the preset keyword indexer;

[0074] Step a42: if it exists, then add an index between the first storage address and the first keyword to the index corresponding to the first keyword;

[0075] Step a43: if it does not exist, create a new index between the first keyword and the first storage address, and store it in a preset keyword indexer.

[0076] In the process of indexing and storing, it is first determined whether the first keyword exists in the preset keyword indexer. If the first keyword already exists in the preset keyword indexer, a new index between the first keyword and the first storage address is directly added to the index corresponding to the first keyword, that is, the storage address is added after the other mapping values ​​(other storage addresses) corresponding to the first keyword. If the first keyword does not exist in the preset keyword indexer, a new index between the first keyword and the first storage address is created and stored in the preset keyword indexer.

[0077] It should be noted that in the prior art, when the keyword is obtained after parsing the data to be stored, if the keyword is used as the key and the data to be stored is stored as the value, if the keyword obtained by parsing is repeated, the previous value will be overwritten, resulting in the loss of the stored data. In this embodiment, the keyword is used as the index value and the storage address is used as the value for mapping and storage. At the same time, when storing, by detecting whether the same keyword exists in the keyword indexer, if so, the corresponding index is directly added after the keyword without overwriting the original value, thereby avoiding the problem of storage data loss in the prior art.

[0078] An embodiment of the present invention provides a data processing method, which obtains data to be stored, stores the data to be stored in a preset key-value memory in the form of a key-value pair, and obtains a corresponding first storage address; parses the data to be stored to obtain a first keyword; establishes an index between the first keyword and the first storage address, and stores it in a preset keyword indexer. In an embodiment of the present invention, while using a key-value pair to store the data to be stored, a storage address is obtained, and an index between the storage address and the keyword is established and stored. Through the above method, a new memory structure is realized, which supports storing data in a preset key-value memory according to a key to obtain a storage address, and also supports parsing data to obtain a keyword, indexing the keyword and the storage address and storing them in a preset keyword indexer. Subsequently, it supports fast search for data according to a key, and also supports fast search for data according to a keyword index. It has good flexibility and versatility and can be applied to different query scenarios. In addition, when searching for data by value in the future, the storage address of the value to be searched can be quickly queried according to the index based on the keyword of the value to be searched, and then the data to be searched can be quickly queried without traversing the entire memory structure, thereby improving the search efficiency and avoiding the problems of low performance and waste of CPU resources due to traversal query in the prior art.

[0079] Furthermore, based on the above-mentioned first embodiment, a second embodiment of the data processing method of the present invention is proposed.

[0080] In this embodiment, after the above step S30, the data processing method further includes:

[0081] Step A, when receiving a data reading request, obtaining data reading information carried in the data reading request;

[0082] In this embodiment, when a data read request is received, data read information carried in the data read request is obtained, wherein the type of the data read information may be a storage address or a keyword.

[0083] Step B11, if the data read information is the second storage address of the data to be searched, then the first target value corresponding to the second storage address is searched from the preset key-value memory and returned to the query end corresponding to the data read request;

[0084] If the data reading information is the second storage address of the data to be searched, the second storage address is directly used as the key to search for the value corresponding to the second storage address from the preset key-value storage device, recorded as the first target value, and returned to the query end corresponding to the data reading request.

[0085] Step B21, if the data read information is a second keyword of the data to be searched, then searching the preset keyword indexer to obtain a target storage address corresponding to the second keyword;

[0086] Step B22, searching the preset key-value memory for a second target value corresponding to the target storage address, and returning the second target value to the query end corresponding to the data read request.

[0087] If the data reading information is the keyword of the data to be searched, recorded as the second target keyword, then the second target keyword is used as the index value, and the value corresponding to the second keyword is first searched from the preset keyword indexer and recorded as the target storage address. Then, the target storage address is used as the key to search from the preset key-value storage device and obtain the value corresponding to the target storage address, recorded as the second target value, and returned to the query end corresponding to the data reading request.

[0088] In addition, it should be noted that, in specific implementation, when searching by keywords, it is also possible to query by means of regular expressions to facilitate fuzzy searches by customers.

[0089] Furthermore, after the data reading is completed, the usage information of the corresponding keyword is updated, and the usage information includes but is not limited to the number of times of use, the usage rate and the usage time, so as to facilitate the subsequent cleaning and recycling.

[0090] In this embodiment, when reading data, not only can the corresponding storage data be directly queried based on the input storage address, but also the corresponding storage address can be first obtained through the index based on the input keyword, and then the corresponding storage data can be queried, that is, while supporting fast search by key, it also supports fast search by index based on keyword. In addition, compared with the prior art, when searching by value, it is necessary to traverse the entire memory structure, and in this embodiment, only the keyword corresponding to the value needs to be known, and the corresponding storage address can be first obtained through the index, and then the corresponding value (i.e., storage data) can be queried through the key-value pair search, thereby improving the search efficiency, avoiding occupying more CPU resources, and improving the utilization of CPU resources.

[0091] Furthermore, based on the above-mentioned first embodiment, a third embodiment of the data processing method of the present invention is proposed.

[0092] In this embodiment, after the above step S30, the data processing method further includes:

[0093] Step C, when receiving a data deletion request, obtaining a third storage address of the data to be deleted according to the data deletion request;

[0094] In this embodiment, when a data deletion request is received, the third storage address of the data to be deleted is obtained according to the data deletion request.

[0095] Step D, searching the preset key-value pair memory for the third target value corresponding to the third storage address and deleting the third target value;

[0096] Then, the third target value corresponding to the third storage address is searched from the preset key-value pair memory and deleted.

[0097] Step E, parsing the third target value to obtain a third keyword;

[0098] The third target value is parsed to obtain a third keyword. The specific parsing method is the same as the parsing method of the data to be stored, and reference may be made to the first embodiment.

[0099] Step F: searching the third keyword from the preset keyword indexer, and deleting the index information between the third keyword and the third storage address.

[0100] Finally, the third keyword is found from the preset keyword indexer, and the index information between the third keyword and the third storage address is deleted.

[0101] In this embodiment, the above method can realize intelligent deletion of data in the data processing memory structure.

[0102] Furthermore, based on the third embodiment described above, a fourth embodiment of the data processing method of the present invention is proposed.

[0103] In this embodiment, the data processing method further includes:

[0104] Step F, when receiving a data modification request, obtaining the target modification data and its fourth storage address, and the modified data according to the data modification request;

[0105] In this embodiment, when a data modification request is received, the target modification data and its fourth storage address, and the modified data are obtained according to the data modification request. The target modification data is the data to be modified, and the modified data is the data to be modified. For example, if A is modified to B, the target modification data is A, and the modified data is B.

[0106] Step G, using the fourth storage address as the third storage address, and executing the steps of: searching the preset key-value pair memory for a third target value corresponding to the third storage address, and deleting the third target value;

[0107] Then, the fourth storage address is used as the third storage address, and the above-mentioned deletion step is performed, that is, the fourth target value corresponding to the fourth storage address is searched from the preset key-value pair memory, and then deleted; the fourth target value is parsed to obtain the fourth keyword; the fourth keyword is searched from the preset keyword indexer, and the index information between the fourth keyword and the fourth storage address is deleted. The specific data deletion process can refer to the above-mentioned third embodiment.

[0108] Step H, after the deletion is completed, the modified data is used as the data to be stored, and the steps are executed: the data to be stored is stored in a preset key-value storage device in the form of a key-value pair, and the corresponding first storage address is obtained.

[0109] After the deletion is completed, the modified data is used as the data to be stored, and the above data storage steps are performed, that is, the modified data is stored in the preset key-value storage in the form of a key-value pair, and the corresponding storage address (recorded as the fifth storage address) is obtained; the modified data is parsed to obtain the corresponding keyword (recorded as the fifth keyword); an index between the fifth keyword and the fifth storage address is established, and stored in the preset keyword indexer. The specific data storage process can refer to the above first embodiment.

[0110] That is to say, during the data modification process, the original data content must be deleted first, and then the modified data content is stored to realize the data modification.

[0111] Furthermore, based on the above-mentioned first to fourth embodiments, a fifth embodiment of the data processing method of the present invention is proposed.

[0112] In this embodiment, the data processing method further includes:

[0113] Step I: when a data change event is detected, the memory occupancy rate and the total memory occupancy rate of the corresponding keyword are updated according to the data change event.

[0114] In this embodiment, when a data change event is detected, the memory occupancy rate and the total memory occupancy rate of the corresponding keyword are updated according to the data change event. The data change event may be generated based on data writing, data deletion or data modification, that is, after the data writing is completed, or after the data is deleted, or after the data is modified, the memory occupancy rate and the total memory occupancy rate of the corresponding keyword will be updated. The memory occupancy rate of a keyword is the memory rate occupied by the information size corresponding to the keyword, and the total memory occupancy rate is the quotient obtained by dividing the current total memory occupancy size by the total memory.

[0115] Furthermore, after the above step I, the data processing method further includes:

[0116] Step J, obtaining the current total memory usage in real time, and determining whether the current total memory usage exceeds a preset threshold;

[0117] Step K: If the current total memory usage exceeds a preset threshold, the stored data is cleaned up according to a preset cleaning rule; wherein the preset cleaning rule includes one or more of the following:

[0118] Clean up the storage data corresponding to the keyword with the largest memory usage, clean up the storage data corresponding to the keyword that has not been used within a preset time period, and clean up the storage data corresponding to the keyword with the lowest usage rate.

[0119] As more and more data is stored, it is easy to cause the memory usage to exceed the standard, thus affecting the normal operation of the program. Therefore, in this embodiment, the total memory occupancy rate is monitored in real time, and then when the memory usage exceeds the standard, the data is cleaned and recycled to ensure the stable and normal operation of the program.

[0120] Specifically, the current total memory occupancy rate is obtained in real time, and it is determined whether the current total memory occupancy rate exceeds the preset threshold. If the current total memory occupancy rate exceeds the preset threshold, the stored data is cleaned up according to the preset cleaning rules; wherein the preset cleaning rules include one or more of the following: cleaning up the stored data corresponding to the keyword with the largest memory occupancy rate, cleaning up the stored data corresponding to the keyword that has not been used within the preset time period, and cleaning up the stored data corresponding to the keyword with the lowest usage rate. Among them, whether it has been used and the usage rate within the preset time period can be determined based on the usage information of the keyword updated after the data is read (including but not limited to the number of times it has been used, the usage rate, and the usage time).

[0121] Of course, it is understood that in a specific embodiment, the cleaning rules are not limited to the above ones, and can be flexibly set according to actual needs. For example, cleaning can also be performed based on the number of times a keyword is used.

[0122] In this embodiment, by recording and updating the changes in the information size of each keyword (i.e., the memory occupancy rate of each keyword) and the changes in the overall size of the memory structure (i.e., the total memory occupancy rate), and when the total memory occupancy rate is monitored to exceed a preset threshold, data cleaning and recovery are performed to ensure stable and normal operation of the program.

[0123] Furthermore, based on the above-mentioned first to fourth embodiments, a sixth embodiment of the data processing method of the present invention is proposed.

[0124] In this embodiment, before the above step of "storing the data to be stored in a preset key-value memory in the form of a key-value pair, and obtaining the corresponding first storage address", the data processing method further includes:

[0125] Step L, obtaining the classification label of the data to be stored, and determining the target key-value storage device and the target keyword indexer according to the classification label;

[0126] In this embodiment, when storing data, it can also be classified and stored by type to facilitate subsequent reading, deletion and modification.

[0127] Specifically, when storing data, the user can set the classification label of the data, wherein the classification label can be classified according to dimensions such as business scenarios and business types, and correspondingly, the key-value storage and keyword indexer are also classified according to the same classification label. After receiving the data to be stored, the server can obtain the classification label of the data to be stored, and determine the target key-value storage and target keyword indexer according to the classification label.

[0128] At this time, the step of "storing the data to be stored in a preset key-value memory in the form of a key-value pair, and obtaining the corresponding first storage address" includes: "storing the data to be stored in the target key-value memory in the form of a key-value pair, and obtaining the corresponding first storage address";

[0129] Step S30 includes: establishing an index between the first keyword and the first storage address, and storing the index in the target keyword indexer.

[0130] Then, the data to be stored is stored in the target key-value storage in the form of a key-value pair, and the corresponding first storage address is obtained. Then, the data to be stored is parsed to obtain a first keyword. After the first keyword is parsed, an index between the first keyword and the first storage address is established and stored in a preset keyword indexer. The specific execution process can refer to the first embodiment above.

[0131] Furthermore, when triggering a data read request, a data deletion request or a data modification request, a relevant classification label can also be input, and then the corresponding keyword indexer or key-value pair storage device is first determined based on the classification label to narrow the scope, and then the relevant operations are performed in the corresponding keyword indexer or key-value pair storage device.

[0132] Taking the data reading scenario as an example, when a data reading request is received, the data reading information and classification label (recorded as the second classification label, to be distinguished from the classification label of the data to be stored) carried in the data reading request are obtained, and then the target keyword indexer and target key-value pair storage are determined according to the second classification label (recorded as the second target keyword indexer and the second target key-value pair storage, respectively, to be distinguished from the target keyword indexer and the target key-value pair storage involved above). If the data reading information is the second storage address of the data to be found, the first target value corresponding to the second storage address is obtained from the target key-value storage device, and returned to the query end corresponding to the data reading request; if the data reading information is the second keyword of the data to be found, the target storage address corresponding to the second keyword is first obtained from the target keyword indexer, and then the second target value corresponding to the target storage address is obtained from the target key-value storage device, and returned to the query end corresponding to the data reading request.

[0133] In this embodiment, when storing data, classified storage is performed based on the classification label of the data to be stored, the corresponding key-value pairs are stored in the corresponding type of key-value storage, and the corresponding index is stored in the corresponding type of keyword indexer. In this way, it is convenient to narrow the search scope in the subsequent data reading, data deletion and data modification operations, thereby improving the corresponding operation efficiency.

[0134] The invention also provides a data processing device.

[0135] Reference Figure 4 , Figure 4 FIG. 1 is a schematic diagram of functional modules of a first embodiment of a data processing device according to the present invention.

[0136] like Figure 4 As shown, the data processing device includes:

[0137] A first acquisition module 10 is used to acquire data to be stored, store the data to be stored in a preset key-value memory in the form of a key-value pair, and acquire a corresponding first storage address;

[0138] A first parsing module 20, used to parse the data to be stored to obtain a first keyword;

[0139] The index building module 30 is used to build an index between the first keyword and the first storage address, and store it in a preset keyword indexer.

[0140] Furthermore, the index building module 30 includes:

[0141] A judging unit, configured to judge whether the first keyword exists in the preset keyword indexer;

[0142] an index adding unit, configured to add an index between the first storage address and the first keyword in the index corresponding to the first keyword, if the index exists;

[0143] An index creation unit is used to create a new index between the first keyword and the first storage address if it does not exist, and store it in a preset keyword indexer.

[0144] Furthermore, the data processing device further includes:

[0145] A second acquisition module, configured to acquire data reading information carried in the data reading request upon receiving the data reading request;

[0146] A first search module, configured to search for a first target value corresponding to the second storage address from the preset key-value memory if the data reading information is a second storage address of the data to be searched, and return the first target value to the query end corresponding to the data reading request;

[0147] A second search module, configured to search for a target storage address corresponding to the second keyword from the preset keyword indexer if the data read information is a second keyword of the data to be searched;

[0148] The third search module is used to search for the second target value corresponding to the target storage address from the preset key-value storage device, and return it to the query end corresponding to the data read request.

[0149] Furthermore, the data processing device further includes:

[0150] A third acquisition module, configured to, upon receiving a data deletion request, acquire a third storage address of the data to be deleted according to the data deletion request;

[0151] A first deleting module, configured to search the preset key-value pair memory for a third target value corresponding to the third storage address and delete the third target value;

[0152] A second parsing module, used for parsing the third target value to obtain a third keyword;

[0153] The second deleting module is used to search the third keyword from the preset keyword indexer and delete the index information between the third keyword and the third storage address.

[0154] Furthermore, the data processing device further includes:

[0155] a fourth acquisition module, configured to acquire, upon receiving a data modification request, target modification data and a fourth storage address thereof, and modified data according to the data modification request;

[0156] A first execution module is used to use the fourth storage address as the third storage address and execute the steps of: searching for a third target value corresponding to the third storage address from the preset key-value pair memory and deleting the third target value;

[0157] The second execution module is used to use the modified data as the data to be stored after the deletion is completed, and execute the steps of: storing the data to be stored in a preset key-value storage device in the form of a key-value pair, and obtaining the corresponding first storage address.

[0158] Furthermore, the data processing device further includes:

[0159] The information update module is used to update the memory occupancy rate and the total memory occupancy rate of the corresponding keyword according to the data change event when a data change event is detected.

[0160] Furthermore, the data processing device further includes:

[0161] A judgment module obtains the current total memory occupancy rate in real time and judges whether the current total memory occupancy rate exceeds a preset threshold;

[0162] A data cleaning module is used to clean up the stored data according to a preset cleaning rule if the current total memory occupancy rate exceeds a preset threshold; wherein the preset cleaning rule includes one or more of the following:

[0163] Clean up the storage data corresponding to the keyword with the largest memory usage, clean up the storage data corresponding to the keyword that has not been used within a preset time period, and clean up the storage data corresponding to the keyword with the lowest usage rate.

[0164] Furthermore, the data processing device further includes:

[0165] A fifth acquisition module, used to acquire the classification label of the data to be stored, and determine the target key-value storage device and the target keyword indexer according to the classification label;

[0166] The first acquisition module 10 is specifically used to: store the data to be stored in the target key-value storage device in the form of a key-value pair, and obtain the corresponding first storage address;

[0167] The index building module 30 is specifically used to: establish an index between the first keyword and the first storage address, and store it in the target keyword indexer.

[0168] Among them, the functional implementation of each module in the above-mentioned data processing device corresponds to each step in the above-mentioned data processing method embodiment, and its functions and implementation processes are no longer repeated here one by one.

[0169] The present invention also provides a computer-readable storage medium having a data processing program stored thereon, and when the data processing program is executed by a processor, the steps of the data processing method described in any of the above embodiments are implemented.

[0170] The specific embodiments of the computer-readable storage medium of the present invention are substantially the same as the embodiments of the above-mentioned data processing method, and are not described in detail herein.

[0171] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0172] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0174] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The data processing method comprises: Acquire the data to be stored, use the storage address of the data to be stored as a key and the data to be stored as a value, store the data to be stored in a preset key-value memory in the form of a key-value pair, and acquire the corresponding first storage address; the preset key-value memory is used to store data content in the form of a key-value pair; the first storage address is the storage address of the data to be stored in the preset key-value memory; parse the data to be stored to obtain a first keyword; An index between the first keyword and the first storage address is established and stored in a preset keyword indexer; the preset keyword indexer is used to search the preset keyword indexer for a target storage address corresponding to the second keyword according to the second keyword in the data read request.

2. The data processing method according to claim 1, characterized in that: The step of establishing an index between the first keyword and the first storage address and storing the index in a preset keyword indexer includes: Determining whether the first keyword exists in the preset keyword indexer; If it exists, then adding an index between the first storage address and the first keyword to the index corresponding to the first keyword; If it does not exist, a new index is created between the first keyword and the first storage address and stored in the preset keyword indexer.

3. The data processing method according to claim 1, characterized in that: The data processing method further includes: When receiving a data reading request, obtaining data reading information carried in the data reading request; If the data read information is the second storage address of the data to be searched, then the first target value corresponding to the second storage address is searched from the preset key-value memory and returned to the query end corresponding to the data read request; If the data read information is a second keyword of the data to be searched, then searching the preset keyword indexer to obtain a target storage address corresponding to the second keyword; A second target value corresponding to the target storage address is searched from the preset key-value memory and returned to the query end corresponding to the data read request.

4. The data processing method according to claim 1, characterized in that: The data processing method further includes: When receiving a data deletion request, obtaining a third storage address of the data to be deleted according to the data deletion request; Searching for a third target value corresponding to the third storage address from the preset key-value pair memory and deleting the third target value; Parsing the third target value to obtain a third keyword; The third keyword is searched from the preset keyword indexer, and index information between the third keyword and the third storage address is deleted.

5. The data processing method according to claim 4, characterized in that: The data processing method further includes: When receiving a data modification request, obtaining target modification data and its fourth storage address, and modified data according to the data modification request; The fourth storage address is used as the third storage address, and the steps are performed: searching the preset key-value pair memory for a third target value corresponding to the third storage address, and deleting the third target value; After the deletion is completed, the modified data is used as the data to be stored, and the steps are performed: the data to be stored is stored in a preset key-value storage device in the form of a key-value pair, and the corresponding first storage address is obtained.

6. The data processing method according to any one of claims 1 to 5, characterized in that: The data processing method further includes: When a data change event is detected, the memory occupancy rate and the total memory occupancy rate of the corresponding keyword are updated according to the data change event.

7. The data processing method according to claim 6, characterized in that: The data processing method further includes: Obtaining the current total memory usage in real time, and determining whether the current total memory usage exceeds a preset threshold; If the current total memory usage exceeds a preset threshold, the stored data is cleaned up according to a preset cleaning rule; wherein the preset cleaning rule includes one or more of the following: Clean up the storage data corresponding to the keyword with the largest memory usage, clean up the storage data corresponding to the keyword that has not been used within a preset time period, and clean up the storage data corresponding to the keyword with the lowest usage rate.

8. The data processing method according to any one of claims 1 to 5, characterized in that: Before the step of storing the data to be stored in a preset key-value memory in the form of a key-value pair and obtaining the corresponding first storage address, the step further includes: Obtaining classification labels of the data to be stored, and determining a target key-value storage device and a target keyword indexer according to the classification labels; The step of storing the data to be stored in a preset key-value memory in the form of a key-value pair and obtaining the corresponding first storage address comprises: Storing the data to be stored in the target key-value storage device in the form of a key-value pair, and obtaining a corresponding first storage address; The step of establishing an index between the first keyword and the first storage address and storing the index in a preset keyword indexer includes: An index is established between the first keyword and the first storage address, and stored in the target keyword indexer.

9. A data processing device, characterized in that: The data processing device comprises: A first acquisition module is used to acquire data to be stored, using the storage address of the data to be stored as a key and the data to be stored as a value, storing the data to be stored in a preset key-value memory in the form of a key-value pair, and acquiring a corresponding first storage address; the preset key-value memory is used to store data content in the form of a key-value pair; the first storage address is the storage address of the data to be stored in the preset key-value memory; A first parsing module, used for parsing the data to be stored to obtain a first keyword; An index building module is used to establish an index between the first keyword and the first storage address, and store it in a preset keyword indexer; the preset keyword indexer is used to search the preset keyword indexer for a target storage address corresponding to the second keyword according to the second keyword in the data reading request.

10. A data processing device, characterized in that: The data processing device comprises: a memory, a processor, and a data processing program stored in the memory and executable on the processor, wherein the data processing program implements the steps of the data processing method according to any one of claims 1 to 8 when executed by the processor.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data processing program, and when the data processing program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 8 are implemented.

Images