Data query method, device, electronic device and storage medium

Abstract

The present invention relates to big data analysis, and provides a data query method, which is applied to electronic devices. The method includes identifying target data from data to be processed, classifying the target data, assigning a preset identifier to each type of target data, obtaining an independent storage space from a first memory as a data pool for storing the target data, recording the location information of each data pool in the first memory, putting the location information into a data storage structure of a second memory, parsing a data query request issued by a client, determining a data query method according to a data type, querying the location information from the data storage structure through a preset identifier, querying the data pool from the first memory, obtaining the target data in the data pool and feeding it back to the client. In addition, the present invention also relates to blockchain technology, and the target data can be stored in a blockchain node. The present invention can avoid memory overflow when performing big data retrieval.

Description

Technical Field

[0001] The present invention relates to big data analysis, and in particular to a data query method, device, electronic device and storage medium. Background Art

[0002] JVM (Java Virtual Machine) is a fictional computer that is implemented by simulating various computer functions on an actual computer. When JVM retrieves data stored on the disk, it stores all eligible data on the disk into the JVM's heap memory. Since the capacity of the JVM's heap memory is small, it is easy to cause memory overflow (i.e. insufficient memory) when retrieving large data. Therefore, how to avoid memory overflow when retrieving large data has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the present invention is to provide a data query method, device, electronic device and storage medium, aiming at how to avoid the problem of memory overflow when performing large data retrieval.

[0004] To achieve the above object, the present invention provides a data query method, which is applied to an electronic device, and includes:

[0005] Classification step: identifying target data from the data to be processed using preset identification rules, classifying the target data, and assigning a unique preset identifier to each type of the target data;

[0006] Processing step: obtaining a preset number of independent storage spaces from the first memory as data pools for storing the target data, each data pool corresponding to a type of target data, and recording the location information of each data pool in the first memory;

[0007] Creating step: storing the location information in the data storage structure of the second memory, creating a first mapping relationship between the location information and its corresponding data pool, and creating a second mapping relationship between the preset identifier and its corresponding location information; and

[0008] Query step: parse the data query request sent by the client, obtain the preset identifier and data type of the target data to be queried, determine the data query method according to the data type, query the location information corresponding to the preset identifier from the data storage structure through the preset identifier, query the data pool corresponding to the location information from the first memory according to the location information, obtain the target data in the data pool and feed it back to the client.

[0009] Preferably, when the data type of the target data to be queried is fragmented data, querying the data storage structure for location information corresponding to the preset identifier through the preset identifier, querying the first memory for a data pool corresponding to the location information according to the location information, and obtaining the target data in the data pool and feeding it back to the client includes:

[0010] Selecting a data pool containing only one data set in the first memory for data query, wherein the location information of the data pool includes a memory address of the data pool in the first memory, a starting address of the target data in the data pool, and an address segment length;

[0011] According to the memory address, querying a data pool corresponding to the memory address from the first memory;

[0012] Determining a starting storage point of the target data from the data pool according to the starting address; and

[0013] Target data of a determined length is obtained according to the length of the address segment and fed back to the client.

[0014] Preferably, when the data type of the target data is set data, querying the location information corresponding to the preset identifier from the data storage structure through the preset identifier, querying the data pool corresponding to the location information from the first memory according to the location information, and obtaining the target data in the data pool and feeding it back to the client includes:

[0015] Selecting a data pool in the first memory that contains more than one target data set to perform data query, wherein the location information of the data pool includes a memory address of the target data pool in the first memory, a starting address of the target data in the data pool, and a total quantity value of the target data;

[0016] According to the memory address, querying a data pool corresponding to the memory address from the first memory;

[0017] Determining a starting storage point of the target data from the data pool according to the starting address; and

[0018] Utilizing a predetermined search algorithm, target data starting from the start address and having a number equal to the total quantity value are obtained from the data pool and fed back to the client.

[0019] Preferably, the search algorithm comprises:

[0020] sorting the target data in the data pool, generating a position sequence of the target data, recording the position sequence number of each target data in the position sequence, and marking the target data corresponding to the starting address as the target sequence number;

[0021] Determine the number of position numbers in the middle position of the position sequence, and when the number of the position number in the middle position is one, use it as the middle sequence number, compare the target sequence number with the middle sequence number, and if the two sequence numbers are the same, take the target data corresponding to the middle sequence number as the starting point, obtain the target data with the total number value, and feed it back to the client;

[0022] If they are inconsistent, determine whether the target sequence number is greater than the middle sequence number. If so, take the middle sequence number as the midpoint, divide all the position sequence numbers in the data pool into a first sequence number set and a second sequence number set, the position sequence numbers in the first sequence number set are smaller than the position sequence numbers in the second sequence number set, compare the position sequence numbers in the middle position in the second sequence number set with the target sequence number, and repeat the above steps until a position sequence number that matches the target sequence number is found from all the position sequence numbers; and

[0023] When there are two position numbers in the middle position, the smaller one is taken as the middle number to be compared with the target number, and the above steps are repeated until a position number that matches the target number is found from all the position numbers.

[0024] Preferably, the method further comprises:

[0025] Monitor whether each location information in the second memory is accessed by the client within a preset time period. If not, call the System.gc() function to query the data pool corresponding to the location information from the first memory, delete the target data stored in the data pool, and release the memory.

[0026] Preferably, the method further comprises:

[0027] The activity value of the target data is calculated periodically or in real time using a preset activity algorithm, and the target data whose activity value is less than a preset threshold is removed from the corresponding data pool to release memory.

[0028] To achieve the above object, the present invention further provides a data query device, the data query device comprising:

[0029] A classification module, used to identify target data from the data to be processed using preset identification rules, classify the target data, and assign a unique preset identifier to each type of the target data;

[0030] a processing module, configured to obtain a preset number of independent storage spaces from the first memory as data pools for storing the target data, each data pool corresponding to a type of target data, and simultaneously record location information of each data pool in the first memory;

[0031] a creation module, configured to store the location information in a data storage structure of the second memory, create a first mapping relationship between the location information and its corresponding data pool, and create a second mapping relationship between the preset identifier and its corresponding location information; and

[0032] The query module is used to parse the data query request sent by the client, obtain the preset identifier and data type of the target data to be queried, determine the data query method according to the data type, query the location information corresponding to the preset identifier from the data storage structure through the preset identifier, query the data pool corresponding to the location information from the first memory according to the location information, obtain the target data in the data pool and feed it back to the client.

[0033] To achieve the above object, the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a data query program, and the data query program implementing the following steps when executed by the processor:

[0034] Classification step: identifying target data from the data to be processed using preset identification rules, classifying the target data, and assigning a unique preset identifier to each type of the target data;

[0035] Processing step: obtaining a preset number of independent storage spaces from the first memory as data pools for storing the target data, each data pool corresponding to a type of target data, and recording the location information of each data pool in the first memory;

[0036] Creating step: storing the location information in the data storage structure of the second memory, creating a first mapping relationship between the location information and its corresponding data pool, and creating a second mapping relationship between the preset identifier and its corresponding location information; and

[0037] Query step: parse the data query request sent by the client, obtain the preset identifier and data type of the target data to be queried, determine the data query method according to the data type, query the location information corresponding to the preset identifier from the data storage structure through the preset identifier, query the data pool corresponding to the location information from the first memory according to the location information, obtain the target data in the data pool and feed it back to the client.

[0038] Preferably, when the data type of the target data to be queried is fragmented data, querying the data storage structure for location information corresponding to the preset identifier through the preset identifier, querying the first memory for a data pool corresponding to the location information according to the location information, and obtaining the target data in the data pool and feeding it back to the client includes:

[0039] Selecting a data pool containing only one data set in the first memory for data query, wherein the location information of the data pool includes a memory address of the data pool in the first memory, a starting address of the target data in the data pool, and an address segment length;

[0040] According to the memory address, querying a data pool corresponding to the memory address from the first memory;

[0041] Determining a starting storage point of the target data from the data pool according to the starting address; and

[0042] Target data of a determined length is obtained according to the length of the address segment and fed back to the client.

[0043] To achieve the above object, the present invention further provides an electronic device, the electronic device comprising:

[0044] at least one processor; and,

[0045] a memory communicatively connected to the at least one processor; wherein,

[0046] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data query method as described above.

[0047] To achieve the above objectives, the present invention further provides a computer-readable storage medium, on which a data query program is stored. The data query program can be executed by one or more processors to implement the steps of the data query method as described above.

[0048] The data query method, device, electronic device and storage medium proposed in the present invention classify the target data by identifying the target data from the data to be processed, and assign a unique preset identifier to each type of target data, obtain an independent storage space from the first memory as a data pool for storing the target data, and record the position information of each data pool in the first memory at the same time, and put each position information into a data storage structure pre-created in the second memory, create a first mapping relationship between the position information and its corresponding data pool, parse the data query request issued by the client, obtain the preset identifier and data type of the target data to be queried, determine the data query method according to the data type, query the data storage structure for the position information corresponding to the preset identifier through the preset identifier, query the data pool corresponding to the position information from the first memory according to the position information, obtain the target data in the data pool and feed it back to the client. The present invention can avoid memory overflow when performing large data retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a flow chart of an electronic device implementing a data query method provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of a module of a data query device provided by an embodiment of the present invention;

[0051] Figure 3 An internal structure diagram of an electronic device for a data query method provided by an embodiment of the present invention.

[0052] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical embodiments and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0054] It should be noted that the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical embodiments between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of the technical embodiments is contradictory or cannot be implemented, it should be deemed that such combination of the technical embodiments does not exist and is not within the scope of protection required by the present invention.

[0055] The present invention provides a data query method. Figure 1 FIG. 1 is a flow chart of a data query method provided by an embodiment of the present invention. The method can be executed by a device, and the device can be implemented by software and / or hardware.

[0056] In this embodiment, the data query method includes:

[0057] S110, identifying target data from the data to be processed using a preset identification rule, classifying the target data, and assigning a unique preset identifier to each category of the target data.

[0058] Taking the securities trading platform as an example, the securities data generated in the securities trading platform is the data to be processed. Securities data has the characteristics of data density, large instantaneous data volume, and the need for real-time query of single-day market-related data. However, not all securities data will be frequently retrieved by users, and most of them are not frequently used or will not be used by users. Therefore, in this embodiment, by adopting preset identification rules, the securities data that is frequently retrieved, that is, the target data, is obtained from the data to be processed, so as to avoid storing all the data to be processed in the off-heap memory without screening, resulting in excessive system resource usage and affecting data processing efficiency.

[0059] Identification rules can be determined based on business needs. For example, in the securities market, the K-line data of the past 30 days, the real-time market conditions of the day, and the transaction data of each transaction are all the focus of users and need to be queried in real time, so these securities data can be used as target data.

[0060] Identification rules can also be distinguished by preset rules. For example, according to the viewing habits of users, when users view market data, they mostly query the most recent market data, such as the latest 1,000 transaction data. You can sort by time and take the 1,000 data closest to the current time as the target data.

[0061] After identifying the target data from the data to be processed, it is necessary to classify it into several categories according to the preset types (such as preset types A, B, and C) corresponding to the target data, and assign a unique preset identifier (such as A, B, and C) to each type of target data. According to the number of target data categories, an equal number of independent storage spaces are requested from the off-heap memory as data pools, and each data pool corresponds to one type of target data.

[0062] S120, obtaining a preset number of independent storage spaces from the first memory as data pools for storing the target data, each data pool corresponding to a type of target data, and recording location information of each data pool in the first memory.

[0063] In this embodiment, the first memory specifically refers to the off-heap memory of the Java virtual machine, and the second memory specifically refers to the on-heap memory of the Java virtual machine. By obtaining a preset number of independent storage spaces from the first memory as a data pool, the target data is stored in the data pool of the first memory. It can reduce the occupancy rate of the second memory of Java and the memory occupancy rate of the old generation of Java, thereby reducing the time and number of times the Java application is executed in the GC (garbage collection) process, ensuring the high response and high throughput of the Java application, and significantly improving the overall performance of the Java application.

[0064] Furthermore, in order to facilitate the user to quickly query the specified data pool from the first memory, in this embodiment, unique location information is allocated to each data pool according to the data pools divided in the off-heap memory.

[0065] S130, storing the location information in the data storage structure of the second memory, creating a first mapping relationship between the location information and its corresponding data pool, and creating a second mapping relationship between the preset identifier and its corresponding location information.

[0066] In this embodiment, the first memory specifically refers to the off-heap memory of the Java virtual machine, and the second memory specifically refers to the on-heap memory of the Java virtual machine. In order to retain the GC (garbage collection) function of Java, the location information of each data pool needs to be placed in the data storage structure of the second memory, such as the heap.

[0067] Since Java's gc is responsible for releasing the memory of the heap, and the heap is created in the second memory rather than the first memory, only by storing the location information of the data pool in the heap of the second memory and creating the first mapping relationship between the location information and its corresponding data pool can the gc function of Java be used to avoid the target data in the first memory from occupying system resources and affecting the normal operation of the system when the target data has not been called for a long time and there is no mechanism for automatically destroying and releasing resources.

[0068] S140, parse the data query request sent by the client, obtain the preset identifier and data type of the target data to be queried, determine the data query method according to the data type, query the location information corresponding to the preset identifier from the data storage structure through the preset identifier, query the data pool corresponding to the location information from the first memory according to the location information, obtain the target data in the data pool and feed it back to the client.

[0069] In this embodiment, the first memory specifically refers to the off-heap memory of the Java virtual machine, and the second memory specifically refers to the on-heap memory of the Java virtual machine. When the electronic device 1 receives the data query request sent by the client, it parses the request to obtain the preset identifier (for example, A) and data type (for example, fragment data) of the data to be queried.

[0070] Determine the data query method according to the data type. Based on the determined data query method, query the location information corresponding to the preset identifier from the data storage structure through the preset identifier, and then query the corresponding data pool from the first memory according to the queried location information, obtain the target data in the data pool and feed it back to the client.

[0071] Specifically, when it is identified that the data type of the target data to be queried is fragmented data, the query step includes:

[0072] Select a data pool containing only one data set in the first memory for data query, wherein the location information of the data pool includes a memory address of the data pool in the first memory, a starting address of the target data in the data pool, and an address segment length;

[0073] According to the memory address, query the data pool corresponding to the memory address from the first memory;

[0074] Determine the starting storage point of the target data from the data pool according to the starting address; and

[0075] The target data of a certain length is obtained according to the length of the address segment and fed back to the client.

[0076] The complexity of this search method is O(1), and it has the advantage of high search efficiency, but it will occupy more system resources. Therefore, this search method should be given priority only when it is determined that the data type of the target data to be queried is fragmented data.

[0077] In another embodiment, when the data type of the target data is set data, the query step further includes:

[0078] Select a data pool containing more than one target data set in the first memory for data query. For example, when the target data is the K-line data of a security, each K-line data corresponds to a target data set, and each K-line data is sorted in chronological order and stored in the data pool. The location information of the data pool includes the memory address of the target data pool in the first memory, the starting address of the target data in the data pool, and the total quantity value of the target data;

[0079] According to the memory address, query the data pool corresponding to the memory address from the first memory;

[0080] Determine the starting storage point of the target data from the data pool according to the starting address; and

[0081] The search algorithm is used to obtain the target data with the starting address as the starting point and the total number value from the data pool and feed it back to the client.

[0082] Specifically, the search algorithm includes:

[0083] Select a data pool containing only one data set in the first memory for data query. For example, when the target data is a certain segment of the K-line data of a security, each K-line data corresponds to a target data set. The location information of the data pool includes the memory address of the data pool in the first memory, the starting address of the target data in the data pool, and the length of the address segment;

[0084] According to the memory address, query the data pool corresponding to the memory address from the first memory;

[0085] Determine the starting storage point of the target data from the data pool according to the starting address; and

[0086] The target data of a certain length is obtained according to the length of the address segment and fed back to the client.

[0087] The complexity of this search method is O(1), and it has the advantage of high search efficiency, but it will occupy more system resources. Therefore, this search method should be given priority only when it is determined that the data type of the target data to be queried is fragmented data.

[0088] In another embodiment, when the data type of the target data is set data, the query step further includes:

[0089] Select a data pool containing more than one target data set in the first memory for data query. For example, when the target data is the K-line data of a security, each K-line data corresponds to a target data set, and each K-line data is sorted in chronological order and stored in the data pool. The location information of the data pool includes the memory address of the target data pool in the first memory, the starting address of the target data in the data pool, and the total quantity value of the target data;

[0090] According to the memory address, query the data pool corresponding to the memory address from the first memory;

[0091] Determine the starting storage point of the target data from the data pool according to the starting address; and

[0092] The search algorithm is used to obtain the target data with the starting address as the starting point and the total number value from the data pool and feed it back to the client.

[0093] Specifically, the search algorithm includes:

[0094] Sort the target data in the data pool, generate a position sequence of the target data (e.g., a, b, c, d, e), record the position number of each target data in the position sequence (e.g., 1, 2, 3, 4, 5), and mark the target data corresponding to the start address as the target sequence number;

[0095] Determine the number of position numbers in the middle position (e.g., c) in the position sequence (e.g., a, b, c, d, e). When the number of position numbers in the middle position is one, use it as the middle number, compare the target number with the middle number, and if the two numbers are the same, use the target data corresponding to the middle number as the starting point, obtain the target data with the total number value, and feed it back to the client;

[0096] If they are inconsistent, determine whether the target sequence number is greater than the middle sequence number. If so, take the middle sequence number as the midpoint, divide all position sequence numbers in the data pool into a first sequence number set and a second sequence number set. The position sequence numbers in the first sequence number set are smaller than the position sequence numbers in the second sequence number set. Compare the position sequence numbers in the middle position of the second sequence number set with the target sequence number, and repeat the above steps until a position sequence number that matches the target sequence number is found from all position sequence numbers; and

[0097] When there are two position numbers in the middle position (for example, the position sequence is a, b, c, d, and the target data in the middle position are b and c respectively), take the smaller one as the middle number and compare it with the target number, and repeat the above steps until a position number that matches the target number is found from all the position numbers.

[0098] The complexity of this search method is O(logN). Compared with ordinary search methods, it has the advantages of fast search efficiency and small system occupation. It is suitable for situations where the amount of data to be processed is huge.

[0099] In another embodiment, the method further comprises the following steps:

[0100] Monitor whether each location information in the second memory is accessed by the client within a preset time period. If not, call the System.gc() function to query the data pool corresponding to the location information from the first memory, delete the target data stored in the data pool, and release the memory.

[0101] In this embodiment, the first memory specifically refers to the off-heap memory of the Java virtual machine, and the second memory specifically refers to the on-heap memory of the Java virtual machine. In order to avoid the situation where the target data in the first memory has not been called for a long time and there is no mechanism for automatically destroying and releasing resources, which will cause the target data to occupy system resources and affect the normal operation of the system. Therefore, by monitoring whether each location information in the second memory has been accessed by the client within a preset time period (for example, one hour), if not, calling the System.gc() function, finding the corresponding data pool from the first memory according to the location information, destroying the target stored in the data pool, and releasing the memory.

[0102] In another embodiment, the method further comprises the following steps:

[0103] The activity value of the target data is calculated periodically or in real time using a preset activity algorithm, and the target data whose activity value is less than a preset threshold is removed from the corresponding data pool to release memory.

[0104] In this embodiment, the first memory specifically refers to the off-heap memory of the Java virtual machine, and the second memory specifically refers to the on-heap memory of the Java virtual machine. In order to avoid the situation where the target data in the first memory has not been called for a long time and there is no mechanism for automatically destroying and releasing resources, causing the target data to occupy system resources and affect the normal operation of the system. Therefore, the activity value of the target data is calculated regularly or in real time by using a preset activity algorithm, and the target data with an activity value less than a preset threshold is removed from the corresponding data pool to release memory.

[0105] Activity algorithms include:

[0106]

[0107] Among them, R is the activity value of the target data, M n is the total number of times the target data is accessed by the client within the preset time period, n is the total preset time period, and n≥2.

[0108] By counting the number of times the target data is accessed by the client in each preset time period (for example, one day), for example, M1=4, M2=5, n=3, the number of times the target data is accessed by the client in each preset time period is substituted into the above activity algorithm to obtain: As the activity value of this type of target data.

[0109] The preset time period depends on the actual situation. For example, when the initial preset time period is "one day", but the target data is only accessed by the client within one hour, the preset time period is changed to a smaller time unit such as "one hour".

[0110] For a detailed description of the above steps, please refer to the following Figure 2 Schematic diagram of program modules of the embodiment of data query program 10 and Figure 3 Description of a method flow chart of an embodiment of a data query method.

[0111] like Figure 2 , which is a functional module diagram of the data query device 100 of the present invention.

[0112] The data query device 100 of the present invention can be installed in an electronic device 1. According to the functions to be implemented, the data query device 100 can include a classification module 110, a processing module 120, a creation module 130 and a query module 140. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and is stored in the memory of the electronic device.

[0113] In this embodiment, the functions of each module / unit are as follows:

[0114] The classification module 110 is used to identify target data from the data to be processed using preset identification rules, classify the target data, and assign a unique preset identifier to each type of the target data.

[0115] Taking the securities trading platform as an example, the securities data generated in the securities trading platform is the data to be processed. Securities data has the characteristics of data density, large instantaneous data volume, and the need for real-time query of single-day market-related data. However, not all securities data will be frequently retrieved by users, and most of them are not frequently used or will not be used by users. Therefore, in this embodiment, by adopting preset identification rules, the securities data that is frequently retrieved, that is, the target data, is obtained from the data to be processed, so as to avoid storing all the data to be processed in the off-heap memory without screening, resulting in excessive system resource usage and affecting data processing efficiency.

[0116] Identification rules can be determined based on business needs. For example, in the securities market, the K-line data of the past 30 days, the real-time market conditions of the day, and the transaction data of each transaction are all the focus of users and need to be queried in real time, so these securities data can be used as target data.

[0117] Identification rules can also be distinguished by preset rules. For example, according to the viewing habits of users, when users view market data, they mostly query the most recent market data, such as the latest 1,000 transaction data. You can sort by time and take the 1,000 data closest to the current time as the target data.

[0118] After identifying the target data from the data to be processed, it is necessary to classify it into several categories according to the preset types (such as preset types A, B, and C) corresponding to the target data, and assign a unique preset identifier (such as A, B, and C) to each type of target data. According to the number of target data categories, an equal number of independent storage spaces are requested from the off-heap memory as data pools, and each data pool corresponds to one type of target data.

[0119] The processing module 120 is used to obtain a preset number of independent storage spaces from the first memory as data pools for storing the target data, each data pool corresponds to a type of target data, and record the location information of each data pool in the first memory.

[0120] In this embodiment, the first memory specifically refers to the off-heap memory of the Java virtual machine, and the second memory specifically refers to the on-heap memory of the Java virtual machine. By obtaining a preset number of independent storage spaces from the first memory as a data pool, the target data is stored in the data pool of the first memory. It can reduce the occupancy rate of the second memory of Java and the memory occupancy rate of the old generation of Java, thereby reducing the time and number of times the Java application is executed in the GC (garbage collection) process, ensuring the high response and high throughput of the Java application, and significantly improving the overall performance of the Java application.

[0121] Furthermore, in order to facilitate the user to quickly query the specified data pool from the first memory, in this embodiment, unique location information is allocated to each data pool according to the data pools divided in the off-heap memory.

[0122] The creation module 130 is used to store the location information in the data storage structure of the second memory, create a first mapping relationship between the location information and its corresponding data pool, and create a second mapping relationship between the preset identifier and its corresponding location information.

[0123] In this embodiment, the first memory specifically refers to the off-heap memory of the Java virtual machine, and the second memory specifically refers to the on-heap memory of the Java virtual machine. In order to retain the GC (garbage collection) function of Java, the location information of each data pool needs to be placed in the data storage structure of the second memory, such as the heap.

[0124] Since Java's gc is responsible for releasing the memory of the heap, and the heap is created in the second memory rather than the first memory, only by storing the location information of the data pool in the heap of the second memory and creating the first mapping relationship between the location information and its corresponding data pool can the gc function of Java be used to avoid the target data in the first memory from occupying system resources and affecting the normal operation of the system when the target data has not been called for a long time and there is no mechanism for automatically destroying and releasing resources.

[0125] The query module 140 is used to parse the data query request issued by the client, obtain the preset identifier and data type of the target data to be queried, determine the data query method according to the data type, query the location information corresponding to the preset identifier from the data storage structure through the preset identifier, query the data pool corresponding to the location information from the first memory according to the location information, obtain the target data in the data pool and feed it back to the client.

[0126] In this embodiment, the first memory specifically refers to the off-heap memory of the Java virtual machine, and the second memory specifically refers to the on-heap memory of the Java virtual machine. When the electronic device 1 receives the data query request sent by the client, it parses the request to obtain the preset identifier (for example, A) and data type (for example, fragment data) of the data to be queried.

[0127] Determine the data query method according to the data type. Based on the determined data query method, query the location information corresponding to the preset identifier from the data storage structure through the preset identifier, and then query the corresponding data pool from the first memory according to the queried location information, obtain the target data in the data pool and feed it back to the client.

[0128] Specifically, when it is identified that the data type of the target data to be queried is fragmented data, the query step includes:

[0129] Select a data pool containing only one data set in the first memory for data query. For example, when the target data is a certain segment of the K-line data of a security, each K-line data corresponds to a target data set. The location information of the data pool includes the memory address of the data pool in the first memory, the starting address of the target data in the data pool, and the length of the address segment;

[0130] According to the memory address, query the data pool corresponding to the memory address from the first memory;

[0131] Determine the starting storage point of the target data from the data pool according to the starting address; and

[0132] The target data of a certain length is obtained according to the length of the address segment and fed back to the client.

[0133] The complexity of this search method is O(1), and it has the advantage of high search efficiency, but it will occupy more system resources. Therefore, this search method should be given priority only when it is determined that the data type of the target data to be queried is fragmented data.

[0134] In another embodiment, when the data type of the target data is set data, the query step further includes:

[0135] Select a data pool containing more than one target data set in the first memory for data query. For example, when the target data is the K-line data of a security, each K-line data corresponds to a target data set, and each K-line data is sorted in chronological order and stored in the data pool. The location information of the data pool includes the memory address of the target data pool in the first memory, the starting address of the target data in the data pool, and the total quantity value of the target data;

[0136] According to the memory address, query the data pool corresponding to the memory address from the first memory;

[0137] Determine the starting storage point of the target data from the data pool according to the starting address; and

[0138] The search algorithm is used to obtain the target data with the starting address as the starting point and the total number value from the data pool and feed it back to the client.

[0139] Specifically, the search algorithm includes:

[0140] Sort the target data in the data pool, generate a position sequence of the target data (e.g., a, b, c, d, e), record the position number of each target data in the position sequence (e.g., 1, 2, 3, 4, 5), and mark the target data corresponding to the start address as the target sequence number;

[0141] Determine the number of position numbers in the middle position (e.g., c) in the position sequence (e.g., a, b, c, d, e). When the number of position numbers in the middle position is one, use it as the middle number, compare the target number with the middle number, and if the two numbers are the same, use the target data corresponding to the middle number as the starting point, obtain the target data with the total number value, and feed it back to the client;

[0142] If they are inconsistent, determine whether the target sequence number is greater than the middle sequence number. If so, take the middle sequence number as the midpoint, divide all position sequence numbers in the data pool into a first sequence number set and a second sequence number set. The position sequence numbers in the first sequence number set are smaller than the position sequence numbers in the second sequence number set. Compare the position sequence numbers in the middle position of the second sequence number set with the target sequence number, and repeat the above steps until a position sequence number that matches the target sequence number is found from all position sequence numbers; and

[0143] When there are two position numbers in the middle position (for example, the position sequence is a, b, c, d, and the target data in the middle position are b and c respectively), take the smaller one as the middle number and compare it with the target number, and repeat the above steps until a position number that matches the target number is found from all the position numbers.

[0144] The complexity of this search method is O(logN). Compared with ordinary search methods, it has the advantages of fast search efficiency and small system occupation. It is suitable for situations where the amount of data to be processed is huge.

[0145] In another embodiment, the program further performs the following steps:

[0146] Monitor whether each location information in the second memory is accessed by the client within a preset time period. If not, call the System.gc() function to query the data pool corresponding to the location information from the first memory, delete the target data stored in the data pool, and release the memory.

[0147] In this embodiment, the first memory specifically refers to the off-heap memory of the Java virtual machine, and the second memory specifically refers to the on-heap memory of the Java virtual machine. In order to avoid the situation where the target data in the first memory has not been called for a long time and there is no mechanism for automatically destroying and releasing resources, which will cause the target data to occupy system resources and affect the normal operation of the system. Therefore, by monitoring whether each location information in the second memory has been accessed by the client within a preset time period (for example, one hour), if not, calling the System.gc() function, finding the corresponding data pool from the first memory according to the location information, destroying the target stored in the data pool, and releasing the memory.

[0148] In another embodiment, the program further performs the following steps:

[0149] The activity value of the target data is calculated periodically or in real time using a preset activity algorithm, and the target data whose activity value is less than a preset threshold is removed from the corresponding data pool to release memory.

[0150] In this embodiment, the first memory specifically refers to the off-heap memory of the Java virtual machine, and the second memory specifically refers to the on-heap memory of the Java virtual machine. In order to avoid the situation where the target data in the first memory has not been called for a long time and there is no mechanism for automatically destroying and releasing resources, causing the target data to occupy system resources and affect the normal operation of the system. Therefore, the activity value of the target data is calculated regularly or in real time by using a preset activity algorithm, and the target data with an activity value less than a preset threshold is removed from the corresponding data pool to release memory.

[0151] Activity algorithms include:

[0152]

[0153] Among them, R is the activity value of the target data, M n is the total number of times the target data is accessed by the client within the preset time period, n is the total preset time period, and n≥2.

[0154] By counting the number of times the target data is accessed by the client in each preset time period (for example, one day), for example, M1=4, M2=5, n=3, the number of times the target data is accessed by the client in each preset time period is substituted into the above activity algorithm to obtain: As the activity value of this type of target data.

[0155] The preset time period depends on the actual situation. For example, when the initial preset time period is "one day", but the target data is only accessed by the client within one hour, the preset time period is changed to a smaller time unit such as "one hour".

[0156] like Figure 3FIG. 1 is a schematic diagram of the structure of an electronic device for implementing the data query method of the present invention.

[0157] The electronic device 1 may include a processor 12 , a memory 11 , and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 12 , such as a data query program 10 .

[0158] Among them, the memory 11 includes at least one type of readable storage medium, and the computer-usable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function, etc.; the storage data area may store data created according to the use of blockchain nodes, etc. The readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the data query program 10, etc., but also be used to temporarily store data that has been output or is to be output.

[0159] The processor 12 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 12 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules (such as data query programs, etc.) stored in the memory 11, and calls data stored in the memory 11, so as to execute various functions of the electronic device 1 and process data.

[0160] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 12, etc.

[0161] Figure 3 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0162] For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 12 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0163] Furthermore, the electronic device 1 may also include a network interface 13. Optionally, the network interface 13 may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0164] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0165] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0166] The data query program 10 stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 12, it can achieve:

[0167] Classification step: identifying target data from the data to be processed using preset identification rules, classifying the target data, and assigning a unique preset identifier to each type of the target data;

[0168] Processing steps: obtaining a preset number of independent storage spaces from the first memory as data pools, each data pool corresponding to a type of target data, and recording the location information of each data pool in the first memory;

[0169] Creating step: putting the location information into the data storage structure of the second memory, creating a first mapping relationship between the location information and its corresponding data pool, and creating a second mapping relationship between the preset identifier and its corresponding location information; and

[0170] Query step: parse the data query request sent by the client, obtain the preset identifier and data type of the target data to be queried, determine the data query method according to the data type, query the location information corresponding to the preset identifier from the data storage structure through the preset identifier, query the data pool corresponding to the location information from the first memory according to the location information, obtain the target data in the data pool and feed it back to the client.

[0171] In another embodiment, the program further performs the following steps:

[0172] Monitor whether each location information in the second memory is accessed by the client within a preset time period. If not, call the System.gc() function to query the data pool corresponding to the location information from the first memory, delete the target data stored in the data pool, and release the memory.

[0173] For a detailed description of the above steps, please refer to the following Figure 2 Schematic diagram of program modules of the embodiment of data query program 10 and Figure 3 Description of a method flow chart of an embodiment of a data query method.

[0174] Reference Figure 2 As shown, Figure 1 Schematic diagram of program modules of an embodiment of a data query program 10. The data query program 10 is divided into multiple modules, which are stored in a memory 11 and executed by a processor 12 to implement the present invention. The module referred to in the present invention refers to a series of computer program instruction segments that can implement specific functions.

[0175] In this embodiment, the data query program 10 includes a classification module 110 , a processing module 120 , a creation module 130 and a query module 140 .

[0176] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0177] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0178] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0180] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0181] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0182] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0183] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A data query method, applied to an electronic device, characterized in that: The method includes: Classification step: identifying target data from the data to be processed using preset identification rules, classifying the target data, and assigning a unique preset identifier to each type of the target data; Processing steps: obtaining a preset number of independent storage spaces from the first memory as data pools for storing the target data, each data pool corresponding to a type of target data, and recording the location information of each data pool in the first memory; Creating step: storing the location information in a data storage structure of a second memory, creating a first mapping relationship between the location information and its corresponding data pool, and creating a second mapping relationship between the preset identifier and its corresponding location information; and Query step: parsing the data query request sent by the client to obtain the preset identifier and data type of the target data to be queried; when the data type of the target data to be queried is fragmented data, selecting a data pool in the first memory that contains only one data set for data query; the location information of the data pool includes the memory address of the data pool in the first memory, the starting address of the target data in the data pool, and the length of the address segment; querying the data pool corresponding to the memory address from the first memory according to the memory address; determining the starting storage point of the target data from the data pool according to the starting address; and obtaining the target data of the determined length according to the length of the address segment and feeding it back to the client; the client; when the data type of the target data is set data, select a data pool in the first memory that contains more than one target data set for data query, the location information of the data pool includes a memory address of the target data pool in the first memory, a starting address of the target data in the data pool and a total quantity value of the target data, query the data pool corresponding to the memory address from the first memory according to the memory address, determine the starting storage point of the target data from the data pool according to the starting address, and, use a predetermined search algorithm to obtain the target data with the starting address as the starting point and the total quantity value from the data pool and feed it back to the client.

2. The data query method according to claim 1, characterized in that: The search algorithm includes: sorting the target data in the data pool, generating a position sequence of the target data, recording the position sequence number of each target data in the position sequence, and marking the target data corresponding to the starting address as the target sequence number; Determine the number of position numbers in the middle position of the position sequence, and when the number of the position number in the middle position is one, use it as the middle sequence number, compare the target sequence number with the middle sequence number, and if the two sequence numbers are the same, take the target data corresponding to the middle sequence number as the starting point, obtain the target data with the total number value, and feed it back to the client; If they are inconsistent, determine whether the target sequence number is greater than the middle sequence number. If so, take the middle sequence number as the midpoint, divide all the position sequence numbers in the data pool into a first sequence number set and a second sequence number set, the position sequence numbers in the first sequence number set are smaller than the position sequence numbers in the second sequence number set, compare the position sequence numbers in the middle position in the second sequence number set with the target sequence number, and repeat the above steps until a position sequence number that matches the target sequence number is found from all the position sequence numbers; and When there are two position sequence numbers in the middle position, the smaller one is taken as the middle sequence number and compared with the target sequence number, and the above steps are repeated until a position sequence number that matches the target sequence number is found from all the position sequence numbers.

3. The data query method according to claim 1, characterized in that: The first memory and the second memory exist in a Java virtual machine, the target data is stored in a blockchain, and the method further includes: Monitor whether each location information in the second memory is accessed by the client within a preset time period. If not, call the System.gc() function to query the data pool corresponding to the location information from the first memory, delete the target data stored in the data pool, and release the memory.

4. The data query method according to claim 1, characterized in that: The first memory and the second memory exist in a Java virtual machine, the target data is stored in a blockchain, and the method further includes: The activity value of the target data is calculated periodically or in real time using a preset activity algorithm, and the target data with an activity value less than a preset threshold is removed from the corresponding data pool to release memory.

5. A data query device, used to implement the data query method according to any one of claims 1 to 4, characterized in that: The data query device comprises: A classification module, used to identify target data from the data to be processed using preset identification rules, classify the target data, and assign a unique preset identifier to each type of the target data; A processing module, used for obtaining a preset number of independent storage spaces from the first memory as data pools for storing the target data, each data pool corresponding to a type of target data, and recording location information of each data pool in the first memory; a creation module, configured to store the location information in a data storage structure of a second memory, create a first mapping relationship between the location information and its corresponding data pool, and create a second mapping relationship between the preset identifier and its corresponding location information; and The query module is used to parse the data query request sent by the client, obtain the preset identifier and data type of the target data to be queried, determine the data query method according to the data type, query the location information corresponding to the preset identifier from the data storage structure through the preset identifier, query the data pool corresponding to the location information from the first memory according to the location information, obtain the target data in the data pool and feed it back to the client.

6. An electronic device, used to implement the data query method according to any one of claims 1 to 4, characterized in that: The electronic device includes a memory and a processor, the memory stores a data query program, and the data query program implements the following steps when executed by the processor: Classification step: identifying target data from the data to be processed using preset identification rules, classifying the target data, and assigning a unique preset identifier to each type of the target data; Processing steps: obtaining a preset number of independent storage spaces from the first memory as data pools for storing the target data, each data pool corresponding to a type of target data, and recording the location information of each data pool in the first memory; Creating step: storing the location information in a data storage structure of a second memory, creating a first mapping relationship between the location information and its corresponding data pool, and creating a second mapping relationship between the preset identifier and its corresponding location information; and Query step: parse the data query request sent by the client, obtain the preset identifier and data type of the target data to be queried, determine the data query method according to the data type, query the location information corresponding to the preset identifier from the data storage structure through the preset identifier, query the data pool corresponding to the location information from the first memory according to the location information, obtain the target data in the data pool and feed it back to the client.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data query method according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data query program, which can be executed by one or more processors to implement the steps of the data query method according to any one of claims 1 to 4.

Images