Data loading method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data loading method and device, electronic equipment and a storage medium, the priority of each piece of data is determined based on historical query information, the data is selected in each data set every time the data is selected, the data in each data set can be loaded to a memory when the spare resource quantity of the memory is sufficient, and the data loading efficiency is improved. The data loading condition of each data set is balanced, and when the data is selected, the data with the highest priority in the data which are not selected in the data set is selected, so that the data with the higher priority can be loaded in the memory, and the use efficiency of the memory is improved under the condition that memory resources are insufficient. Besides, the data loaded to the memory is loaded according to the historical query information of the data, that is, the possibility of next query of the data is predicted based on the historical query information, and the data with relatively high possibility of next query is loaded to the memory, so that the overall query time consumption can be effectively reduced, the query efficiency is improved, and the query efficiency is improved. And increasing a daily query data upper limit.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data loading method, device, electronic device and storage medium. Background Art

[0002] In large-scale data, the amount of data can reach billions or even tens of billions. The data is divided into multiple data sets, each of which includes at least one piece of data. When retrieving data, the data needs to be loaded into the memory. However, because the amount of data is too large and the existing hardware resources are not sufficient to load all data sets into the memory, how to improve memory usage efficiency when memory resources are tight has become an issue of increasing concern. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a data loading method, device, electronic device and storage medium to improve memory usage efficiency when memory resources are tight. The specific technical solution is as follows:

[0004] The present application provides a data loading method, the method comprising:

[0005] Acquire historical query information of data of each data set; each of the data sets includes at least one piece of data;

[0006] Determining the priority of each of the data respectively according to the historical query information;

[0007] For each data set, select the data with the highest priority from the data that have not been selected in the data set;

[0008] Acquire the current free resource amount of the memory, and load the currently selected data if the current free resource amount is not less than the data amount of the currently selected data;

[0009] Return to the execution step: for each data set, select the data with the highest priority from the data that has not been selected in the data set, until the current free resource amount is insufficient or the data in each of the data sets are all loaded.

[0010] In an optional implementation, the data includes subsystem information and environment information, wherein the subsystem information represents the source of the data, and the environment information represents the application scenario of the data;

[0011] The method further comprises:

[0012] Determine the subsystem grouping conditions and environment grouping conditions;

[0013] According to the subsystem grouping condition and the environment grouping condition, each data is grouped based on the subsystem information and environment information of each data to obtain each data set.

[0014] In an optional embodiment, the historical query information includes at least one of the following parameters: query frequency, query time, and data loading time, wherein the higher the query frequency of the data, the higher the priority of the data; the shorter the time from the last query time to the current time, the higher the priority of the data; and the shorter the data loading time, the higher the priority of the data.

[0015] In an optional implementation, the method further includes:

[0016] Based on the query time, determine an average query interval time for the data;

[0017] Determining the priority of each data according to the historical query information includes:

[0018] Determining an initial priority of data according to the historical query information;

[0019] The priority of the data is determined according to the initial priority of the data and the average query interval time of the data.

[0020] In an optional implementation manner, determining the average query interval time of data based on the query time includes:

[0021] After each query is completed, the time interval between two adjacent queries is recorded in real time, and the query interval time is calculated based on each time interval;

[0022] The current average query interval is calculated according to the following formula:

[0023] avg_query_interval'=avg_query_interval*c2

[0024] avg_query_interval is the query interval, c2 is the preset threshold coefficient, and avg_query_interval' is the average query interval;

[0025] and / or

[0026] In the case that the current free resource amount is not less than the data amount of each currently selected data, after loading each currently selected data, the method further includes:

[0027] Calculate the current memory availability information according to the following formula:

[0028] free_mem=total_mem*c1–used_mem

[0029] Where free_mem is the current memory free information, total_mem is the total system memory, used_mem is the memory information occupied by the system, and c1 is the preset threshold coefficient.

[0030] In an optional implementation manner, the obtaining of the current free resource amount of the memory, and loading the currently selected data when the current free resource amount is not less than the data amount of each currently selected data, includes:

[0031] Sort the currently selected data in descending order of priority to obtain a priority data queue;

[0032] Selecting the data at the head of the queue in the priority data queue to obtain first data, wherein the selected first data is removed from the priority data queue;

[0033] Acquire a current free resource amount of the memory, and load the first data if the current free resource amount is not less than the data amount of the first data;

[0034] Return to the execution step: select the data at the head of the queue in the priority data queue to obtain the first data, until all the currently selected data are loaded.

[0035] In an optional implementation manner, the acquiring the current free resource amount of the memory and loading the first data when the current free resource amount is not less than the data amount of the first data includes:

[0036] Acquire the current free resource amount of the memory, and if the current free resource amount is not less than the data amount of the first data, compare the first data with the data loaded in the memory, and if the comparison is unsuccessful, load the first data;

[0037] If the comparison is successful, it is determined that the first data has been loaded into the memory.

[0038] In an optional implementation, the method further includes:

[0039] In a case where the current free resource amount is less than the data amount of the first data, selecting target data from a target data set and releasing the memory resource amount occupied by the target data, the target data set including data that has been loaded into the memory and is not in the priority data queue, and / or data that has been loaded into the memory and has a priority lower than that of the first data;

[0040] When the data volume of the first data is smaller than the current free resource volume, load the first data; when the data volume of the first data is not smaller than the current free resource volume, return to step: select the target data in the target data set and release the memory resources occupied by the target data set to continue execution until the data volume of the first data is smaller than the current free resource volume, or all the data in the target data set has been released.

[0041] In an optional implementation manner, selecting target data from the target data set and releasing memory resources occupied by the target data includes:

[0042] The data that has been loaded into the memory and is not in the priority data queue is preferentially selected as the target data.

[0043] In an optional implementation manner, the preferentially selecting the data that has been loaded into the memory and is not in the priority data queue as the target data includes:

[0044] The data that has been loaded into the memory and is not in the priority data queue is taken as target data in order from large to small in terms of the amount of memory resources occupied.

[0045] In an optional implementation, the historical query information includes: query time and data loading time, and for each piece of data, determining the priority of the data according to the historical query information of the data includes:

[0046] Prioritize your data according to the following formula;

[0047] initial_order(i)

[0048] =query_time_order(i)×c3(i)+query_frequency_order(i)×c4(i)

[0049] +load_cost_order(i)×c4(i)

[0050] Among them, query_time_order(i) is the ranking of data i in the entire data set in reverse order of query time; query_frequency_order is the ranking of data i in the entire data set in ascending order of query frequency; load_cost_order(i) is the ranking of data i in the entire data set in reverse order of load time; c3, c4, c5 are preset weight values, and initial_order(i) is the priority of data i.

[0051] The present application also provides a data loading device, the device comprising:

[0052] An acquisition module, used to acquire historical query information of data of each data set; each of the data sets includes at least one piece of data;

[0053] A determination module, used to determine the priority of each data according to the historical query information;

[0054] A selection module, for selecting, for each data set, data with the highest priority from data that has not been selected in the data set;

[0055] A loading module, used for obtaining the current free resource amount of the memory, and loading each currently selected data when the current free resource amount is not less than the data amount of each currently selected data;

[0056] The processing module is used to return to the execution step: for each data set, the data with the highest priority is selected from the data that has not been selected in the data set, until the current free resource amount is insufficient or the data in each of the data sets are loaded.

[0057] In an optional implementation manner, the data includes subsystem information and environment information, wherein the subsystem information represents the source of the data, and the environment information represents the application scenario of the data;

[0058] The device also includes:

[0059] A grouping determination module, used to determine subsystem grouping conditions and environment grouping conditions;

[0060] The grouping module is used to group the data according to the subsystem grouping condition and the environment grouping condition, based on the subsystem information and environment information of each data, to obtain each data set.

[0061] In an optional embodiment, the historical query information includes at least one of the following parameters: query frequency, query time, and data loading time, wherein the higher the query frequency of the data, the higher the priority of the data; the shorter the time from the last query time to the current time, the higher the priority of the data; and the shorter the data loading time, the higher the priority of the data.

[0062] In an optional embodiment, the device further comprises:

[0063] An average query interval time determination module, used to determine an average query interval time of data based on the query time;

[0064] The determination module is specifically used for:

[0065] Determining an initial priority of data according to the historical query information;

[0066] The priority of the data is determined according to the initial priority of the data and the average query interval time of the data.

[0067] In an optional implementation manner, the average query interval time determination module is specifically used to:

[0068] After each query is completed, the time interval between two adjacent queries is recorded in real time, and the query interval time is calculated based on each time interval;

[0069] The current average query interval is calculated according to the following formula:

[0070] avg_query_interval'=avg_query_interval*c2

[0071] avg_query_interval is the query interval, c2 is the preset threshold coefficient, and avg_query_interval' is the average query interval;

[0072] and / or

[0073] The device also includes:

[0074] The memory free space calculation module is used to calculate the current memory free space information according to the following formula after loading the currently selected data when the current free resource amount is not less than the data amount of each currently selected data:

[0075] free_mem=total_mem*c1–used_mem

[0076] Where free_mem is the current memory free information, total_mem is the total system memory, used_mem is the memory information occupied by the system, and c1 is the preset threshold coefficient.

[0077] In an optional implementation manner, the loading module is specifically used to:

[0078] Sort the currently selected data in descending order of priority to obtain a priority data queue;

[0079] Selecting the data at the head of the queue in the priority data queue to obtain first data, wherein the selected first data is removed from the priority data queue;

[0080] Acquire a current free resource amount of the memory, and load the first data if the current free resource amount is not less than the data amount of the first data;

[0081] Return to the execution step: select the data at the head of the queue in the priority data queue to obtain the first data, until all the currently selected data are loaded.

[0082] In an optional implementation manner, the loading module is specifically used to:

[0083] Acquire the current free resource amount of the memory, and if the current free resource amount is not less than the data amount of the first data, compare the first data with the data loaded in the memory, and if the comparison is unsuccessful, load the first data;

[0084] If the comparison is successful, it is determined that the first data has been loaded into the memory.

[0085] In an optional embodiment, the device further comprises:

[0086] Release module for:

[0087] In a case where the current free resource amount is less than the data amount of the first data, selecting target data from a target data set and releasing the memory resource amount occupied by the target data, the target data set including data that has been loaded into the memory and is not in the priority data queue, and / or data that has been loaded into the memory and has a priority lower than that of the first data;

[0088] When the data volume of the first data is smaller than the current free resource volume, load the first data; when the data volume of the first data is not smaller than the current free resource volume, return to step: select the target data in the target data set and release the memory resources occupied by the target data set to continue execution until the data volume of the first data is smaller than the current free resource volume, or all the data in the target data set has been released.

[0089] In an optional implementation, the release module is specifically used to:

[0090] The data that has been loaded into the memory and is not in the priority data queue is preferentially selected as the target data.

[0091] In an optional implementation, the release module is specifically used to:

[0092] The data that has been loaded into the memory and is not in the priority data queue is taken as target data in order from large to small in terms of the amount of memory resources occupied.

[0093] In an optional implementation manner, the historical query information includes: query time and data loading time, and the determination module is specifically used to:

[0094] Prioritize your data according to the following formula;

[0095] initial_order(i)

[0096] =query_time_order(i)×c3(i)+query_frequency_order(i)×c4(i)

[0097] +load_cost_order(i)×c4(i)

[0098] Among them, query_time_order(i) is the ranking of data i in the entire data set in reverse order of query time; query_frequency_order is the ranking of data i in the entire data set in ascending order of query frequency; load_cost_order(i) is the ranking of data i in the entire data set in reverse order of load time; c3, c4, c5 are preset weight values, and initial_order(i) is the priority of data i.

[0099] The present application also provides an electronic device, including:

[0100] Memory, used to store computer programs;

[0101] The processor is used to implement any of the above-mentioned data loading methods when executing the program stored in the memory.

[0102] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned data loading methods is implemented.

[0103] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned data loading methods.

[0104] Beneficial effects of the embodiments of the present application:

[0105] The embodiments of the present application provide a data loading method, device, electronic device and storage medium, which determines the priority of each data based on historical query information. Each time data is selected, data is selected from each data set. In this way, when there is sufficient free memory resources, the data in each data set can be loaded into the memory to balance the data loading of each data set. When data is selected, the data with the highest priority among the data that has not been selected in the data set is selected, so that the memory can load the data with higher priority, and improve the memory usage efficiency when the memory resources are tight. In addition, the data loaded into the memory is loaded according to the historical query information of the data, that is, the possibility of querying the data next time is predicted based on the historical query information, and the data with a higher possibility of the next query is loaded into the memory, which can effectively reduce the overall query time, improve the query efficiency, and increase the upper limit of daily query data.

[0106] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0108] Figure 1 A first flow chart of the data loading method provided in the embodiment of the present application;

[0109] Figure 2-1 A second flow chart of the data loading method provided in the embodiment of the present application;

[0110] Figure 2-2 A third flow chart of the data loading method provided in the embodiment of the present application;

[0111] Figure 3 A schematic diagram of the structure of a data loading device provided in an embodiment of the present application;

[0112] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0113] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0114] like Figure 1 As shown, Figure 1 A first flow chart of a data loading method provided in an embodiment of the present application, the method comprising:

[0115] S110, obtaining historical query information of data of each data set; each of the above data sets includes at least one piece of data;

[0116] S120, determining the priority of each of the above data according to the above historical query information;

[0117] S130, for each data set, selecting data with the highest priority from data that has not been selected in the data set;

[0118] S140, obtaining the current free resource amount of the memory, and loading the currently selected data when the current free resource amount is not less than the data amount of each currently selected data;

[0119] S150, returning to step S130, until the current free resource amount is insufficient or the data in each of the data sets are loaded completely.

[0120] The data may be vector data, wherein vector data is a method of processing data (images / texts) and other materials through embedding, that is, mapping the data into one or more low-dimensional, dense, and computer-recognizable vector data.

[0121] When searching for data, the data may include information in multiple dimensions.

[0122] In an optional implementation manner, the data includes subsystem information and environment information, wherein the subsystem information represents the source of the data, and the environment information represents the application scenario of the data;

[0123] The method further comprises:

[0124] Determine the subsystem grouping conditions and environment grouping conditions;

[0125] According to the subsystem grouping condition and the environment grouping condition, each data is grouped based on the subsystem information and environment information of each data to obtain each data set.

[0126] The data includes two types of information: subsystem information and environmental information. The subsystem information represents the source of the data, and the environmental information represents the application scenario of the data. Data can be grouped based on information in two dimensions, namely, subsystem information and environment information, to obtain a data set. For example, the subsystem grouping condition and the environment grouping condition can be determined first. According to the subsystem grouping condition and the environment grouping condition, the data set can be divided into multiple subsystems and multiple environments. Each data set can include at least one data slice, and at least one data slice represents at least one data. Assuming there are 3 subsystems and 2 environments, if the subsystem is identified as s and the environment is e, the data set can be divided into 6 data sets. For example, the data set can be identified as: s1e1, s1e2, s2e1, s2e1, s3e1, s3e2. If each data set includes 2 data, all the data can be identified as s1e1-1, s1e1-2, s1e2-1, s1e2-2, s2e1-1, s2e1-2, s2e2-1, s2e2-2, s3e1-1, s3e1-2, s3e2-1, s3e2-2, a total of 12 data. Among them, s1e1-1 represents data 1 of subsystem s1 environment e1.

[0127] Each time a vector search is performed, the subsystem and environment conditions are included. Assuming that the query subsystems are s1 and s2 and the environment is e1, the query will only be performed in the four data sets s1e1-1, s1e1-2, s2e1-1, and s2e1-2.

[0128] According to the query conditions, including subsystem, environment, query time, etc., which are associated with the data set information (data volume, loading time, and memory required for loading), the query information can be recorded after each query. The query information may include query frequency, query time, data loading time, etc., as shown in Table 1.

[0129] Table 1

[0130]

[0131] The priority of each data is determined based on the historical query information of the data. In an optional implementation, the query frequency, query time, and data loading time are used, wherein the higher the query frequency of the data, the higher the priority of the data; the shorter the time from the last query time to the current time, the higher the priority of the data; and the shorter the data loading time, the higher the priority of the data.

[0132] Historical query information includes at least one of the following parameters: query frequency, query time, and data loading time. The query frequency represents the frequency of querying data. Based on the query time, the length of time from the last query time to the current time can be determined, and the average query time interval of the data can also be determined. The data loading time is the time required to load data from the data set into the memory.

[0133] In an optional implementation, the historical query information includes: query time and data loading time. For each piece of data, determining the priority of the data according to the historical query information of the data includes:

[0134] Prioritize your data according to the following formula;

[0135] initial_order(i)

[0136] =query_time_order(i)×c3(i)+query_frequency_order(i)×c4(i)+load_cost_order(i)×c4(i)

[0137] Among them, query_time_order(i) is the ranking of data i in the entire data set in reverse order of the last query time; query_frequency_order is the ranking of data i in the entire data set in ascending order of query frequency; load_cost_order(i) is the ranking of data i in the entire data set in reverse order of load time; c3, c4, c5 are preset weight values, and initial_order(i) is the priority of data i.

[0138] In an optional embodiment, the method further comprises:

[0139] Based on the query time, determine the average query interval of the data;

[0140] Determining the priority of each data according to the historical query information includes:

[0141] Determine the initial priority of the data according to at least one of the query frequency of the data, the last query time, and the data loading time;

[0142] The priority of the data is determined according to the initial priority of the data and the average query interval time of the data.

[0143] After each query is completed, the time interval between two adjacent queries is recorded in real time, and the average query interval and current memory free information are updated, where the current memory free information is the free memory minus a certain amount of memory occupied by the system.

[0144] Calculate the current memory availability information according to the following formula:

[0145] free_mem=total_mem*c1–used_mem

[0146] Where free_mem is the current memory free information, total_mem is the total system memory, used_mem is the memory information occupied by the system, and c1 is the preset threshold coefficient.

[0147] The current average query interval is calculated according to the following formula:

[0148] avg_query_interval'=avg_query_interval*c2

[0149] avg_query_interval is the calculated query interval, c2 is the preset threshold coefficient, and avg_query_interval' is the average query interval.

[0150] The data priority is first determined based on at least one of the data query frequency, the last query time, and the data loading time, and then the data priority is secondarily screened based on the average query interval time to determine the data priority. For example, if the average query interval time of the data is not less than the preset query interval time, the data priority is determined to be 0. That is, after the data priority is first determined based on at least one of the data query frequency, the last query time, and the data loading time, the data with an average query interval time not less than the preset query interval time is screened out, and the data with an average query interval time not less than the preset query interval time is not preferentially loaded. In this way, the data with a shorter average query interval time can be preferentially loaded.

[0151] In each data set, the data with the highest priority among the data that have not been selected will be selected, including a total of 4 data sets: data set s1e1-1, data set s1e1-2, data set s2e1-1, data set s2e1-2. Each of the 4 data sets includes at least 1 data.

[0152] s1e1-1-A is the data with the highest priority selected from the data that has not been selected in the data set s1e1-1, s1e1-2-B is the data with the highest priority selected from the data set s1e1-2, s1e2-1-C is the data with the highest priority selected from the data set s1e2-1, and s1e2-2-D is the data with the highest priority selected from the data set s1e2-2.

[0153] The amount of data represents the amount of memory required for data loading, that is, the amount of memory required to load the data from the data set into the memory. The data with the highest priority among the data that have not been selected in the four data sets are s1e1-1-A, s1e1-2-B, s1e2-1-C, and s1e2-2-D. The current amount of free resources in the memory is obtained, and the current amount of free resources in the memory is compared with the amount of data of each currently selected data. If the current amount of free resources is not less than the amount of data of each currently selected data, it means that the current amount of free resources is sufficient for each data currently selected by the base station, then the currently selected data is loaded, and then the data with the highest priority among the data that have not been selected in the data set is continued to be selected, and it is continued to be determined whether to load until the current amount of free resources is insufficient or the data in each data set is loaded.

[0154] By determining the priority of each data based on historical query information, each time data is selected, data is selected from each data set. In this way, when there is sufficient free memory resources, the data in each data set can be loaded into the memory to balance the data loading of each data set. When selecting data, the data with the highest priority that has not been selected in the data set is selected, so that the memory can load the data with higher priority, which improves the efficiency of memory use when memory resources are tight. In addition, the data loaded into the memory is loaded according to the historical query information of the data, that is, the possibility of querying the data next time is predicted based on the historical query information, and the data with a higher possibility of the next query is loaded into the memory, which can effectively reduce the overall query time, improve query efficiency, and increase the upper limit of daily query data.

[0155] like Figure 2-1 As shown, in an optional implementation manner, step S140 can be specifically implemented by the following steps:

[0156] S1401, sorting the currently selected data in descending order of priority to obtain a priority data queue;

[0157] S1402, selecting the data at the head of the queue in the priority data queue to obtain first data, wherein the selected first data is removed from the priority data queue;

[0158] S1403, obtaining a current free resource amount of the memory, and loading the first data if the current free resource amount is not less than the data amount of the first data;

[0159] S1404, returning to step S1402, until all currently selected data are loaded.

[0160] After selecting each data, it is necessary to traverse each data in order from high to low priority to determine whether the data can be loaded into the memory.

[0161] For example, there are 4 data sets in total: data set s1e1-1, data set s1e1-2, data set s2e1-1, and data set s2e1-2. Each of the 4 data sets includes at least 1 data.

[0162] s1e1-1-A is the data with the highest priority selected from the data that has not been selected in the data set s1e1-1, s1e1-2-B is the data with the highest priority selected from the data set s1e1-2, s1e2-1-C is the data with the highest priority selected from the data set s1e2-1, and s1e2-2-D is the data with the highest priority selected from the data set s1e2-2.

[0163] The data with the highest priority among the data that have not been selected in the four data sets are s1e1-1-A, s1e1-2-B, s1e2-1-C, and s1e2-2-D.

[0164] Sort the data s1e1-1-A, s1e1-2-B, s1e2-1-C, s1e2-2-D in descending order of priority. If the sorting result is s1e1-2-B, s1e1-1-A, s1e2-1-C, s1e2-2-D, the priority data queue is s1e1-2-B, s1e1-1-A, s1e2-1-C, s1e2-2-D.

[0165] First, take s1e1-2-B as the first data, obtain the current free resources of the memory, and determine whether the current free resources of the memory are less than the data volume of the first data. If the current free resources are not less than the data volume of the first data, it means that the current free resources are sufficient to load the first data, and then load the first data into the memory. Because the first data selected from the priority data queue is removed from the priority data queue, after s1e1-2-B is selected as the first data, s1e1-1-A is the data at the head of the priority data queue. Therefore, after s1e1-2-B is loaded into the memory, continue to take s1e1-1-A as the first data, and then determine whether the current free resources of the memory are less than the data volume of s1e1-1-A. If the current free resources are not less than the data volume of s1e1-1-A, load s1e1-1-A until all currently selected data are loaded or the memory is exhausted.

[0166] like Figure 2-2As shown, in an optional implementation manner, step S1403 can be specifically implemented by the following steps:

[0167] S14031, obtaining a current free resource amount of the memory, and when the current free resource amount is not less than a data amount of the first data, comparing the first data with the data loaded into the memory;

[0168] S14032, determine whether the comparison is successful, if not, execute step S14033, if yes, execute step S14034;

[0169] S14033, loading the first data;

[0170] S14034, determine that the first data has been loaded into the memory.

[0171] After selecting the first data, first compare the first data with the data loaded into the memory. If the comparison is unsuccessful, it means that the first data has not been loaded into the memory and the first data can be loaded. If the comparison is successful, it is determined that the first data has been loaded into the memory and does not need to be loaded again, thereby avoiding repeated loading and improving loading efficiency.

[0172] In an optional implementation, the above method further includes:

[0173] In the case where the current free resource amount is less than the data amount of the first data, target data is selected from the target data set and the memory resource amount occupied by the target data is released, the target data set includes data that has been loaded into the memory and is not in the priority data queue, and / or data that has been loaded into the memory and has a priority lower than the priority of the first data;

[0174] When the data volume of the above-mentioned first data is smaller than the current amount of free resources, load the above-mentioned first data; when the data volume of the above-mentioned first data is not smaller than the current amount of free resources, return to step: select the target data in the target data set and release the memory resources occupied by the above-mentioned target data to continue execution until the data volume of the above-mentioned first data is smaller than the current amount of free resources, or all the data in the above-mentioned target data set have been released.

[0175] After the last data query, data may have been loaded into the memory, and the loaded data may include data in the priority data queue and data in the non-priority data queue. The data in the non-priority data queue is data that is not in the priority data queue. For the first data, the priority of the data in the loaded priority data queue may be lower than the priority of the first data. Therefore, data that has been loaded into the above-mentioned memory and is not in the above-mentioned priority data queue, and data that has been loaded into the memory and has a priority lower than the priority of the first data can be added to the target data set. When the current amount of free resources is insufficient to load the data amount of the first data, the data in the target data set can be released.

[0176] After selecting the target data from the target data set, the memory resources occupied by the target data are released, and then the current free resources are updated to determine whether the data volume of the first data is less than the current free resources. If so, the first data is loaded. If the data volume of the first data is still not less than the current free resources, it means that the current free resources are still insufficient to load the first data. Then other data in the target data set will continue to be released until the data volume of the first data is less than the current free resources, or all the data in the target data set has been released. This allows the memory to load data with higher priority, effectively reducing the overall query time and improving memory usage efficiency when memory resources are tight.

[0177] In an optional implementation manner, the selecting target data from the target data set and releasing the memory resources occupied by the target data include:

[0178] The data that has been loaded into the memory and is not in the priority data queue is preferentially selected as the target data.

[0179] The data that has been loaded into the memory and is in the non-priority data queue is less likely to be the data for the next query. In order to avoid related data occupying memory, such data can be prioritized as target data to release the memory resources occupied by this data so that the memory can load higher priority data. In the case of tight memory resources, the overall query time can be effectively reduced and memory usage efficiency can be improved.

[0180] In an optional implementation manner, the data that has been loaded into the memory and is not in the priority data queue is preferentially selected as the target data, including:

[0181] The data that has been loaded into the memory and is not in the priority data queue is taken as target data in order from large to small in terms of the amount of memory resources occupied.

[0182] When releasing data that has been loaded into the memory and is in a non-priority data queue, the data can be released one by one in order from large to small in terms of the amount of memory resources occupied by the data. In the case of tight memory resources, the memory release efficiency can be improved, thereby improving the efficiency of data loading.

[0183] like Figure 3 As shown, the embodiment of the present application also provides a data loading device, the device comprising:

[0184] An acquisition module 310 is used to acquire historical query information of data of each data set; each of the above data sets includes at least one piece of data;

[0185] A determination module 320, configured to determine the priority of each of the above data according to the above historical query information;

[0186] A selection module 330 is used to select, for each data set, data with the highest priority from data that have not been selected in the data set;

[0187] The loading module 340 is used to obtain the current free resource amount of the memory, and load the currently selected data when the current free resource amount is not less than the data amount of the currently selected data;

[0188] The processing module 350 is used to return to the execution step: for each data set, select the data with the highest priority from the data that has not been selected in the data set, until the current free resource amount is insufficient or the data in each of the data sets are loaded.

[0189] In an optional implementation, the data includes subsystem information and environment information, wherein the subsystem information represents the source of the data, and the environment information represents the application scenario of the data;

[0190] The device also includes:

[0191] A grouping determination module, used to determine subsystem grouping conditions and environment grouping conditions;

[0192] The grouping module is used to group the data according to the subsystem grouping condition and the environment grouping condition, based on the subsystem information and environment information of each data, to obtain each data set.

[0193] In an optional embodiment, the historical query information includes at least one of the following parameters: query frequency, query time, and data loading time, wherein the higher the query frequency of the data, the higher the priority of the data; the shorter the time from the last query time to the current time, the higher the priority of the data; and the shorter the data loading time, the higher the priority of the data.

[0194] In an optional embodiment, the device further comprises:

[0195] An average query interval time determination module, used to determine an average query interval time of data based on the query time;

[0196] The determination module is specifically used for:

[0197] Determining an initial priority of data according to the historical query information;

[0198] The priority of the data is determined according to the initial priority of the data and the average query interval time of the data.

[0199] In an optional implementation manner, the average query interval time determination module is specifically used to:

[0200] After each query is completed, the time interval between two adjacent queries is recorded in real time, and the query interval time is calculated based on each time interval;

[0201] The current average query interval is calculated according to the following formula:

[0202] avg_query_interval'=avg_query_interval*c2

[0203] avg_query_interval is the query interval, c2 is the preset threshold coefficient, and avg_query_interval' is the average query interval;

[0204] and / or

[0205] The device also includes:

[0206] The memory free space calculation module is used to calculate the current memory free space information according to the following formula after loading the currently selected data when the current free resource amount is not less than the data amount of each currently selected data:

[0207] free_mem=total_mem*c1–used_mem

[0208] Where free_mem is the current memory free information, total_mem is the total system memory, used_mem is the memory information occupied by the system, and c1 is the preset threshold coefficient.

[0209] In an optional implementation manner, the loading module 340 is specifically used for:

[0210] Sort the currently selected data in descending order of priority to obtain a priority data queue;

[0211] Selecting the data at the head of the queue in the priority data queue to obtain the first data, wherein the selected first data is removed from the priority data queue;

[0212] Acquire the current free resource amount of the memory, and load the first data if the current free resource amount is not less than the data amount of the first data;

[0213] Return to the execution step: select the data at the head of the queue in the above priority data queue to obtain the first data, until all the currently selected data are loaded.

[0214] In an optional implementation manner, the loading module 340 is specifically used for:

[0215] Obtaining the current free resource amount of the memory, and when the current free resource amount is not less than the data amount of the first data, comparing the first data with the data loaded in the memory, and when the comparison is unsuccessful, loading the first data;

[0216] When the comparison is successful, it is determined that the first data has been loaded into the memory.

[0217] In an optional embodiment, the above device further includes:

[0218] Release module for:

[0219] In the case where the current free resource amount is less than the data amount of the first data, target data is selected from the target data set and the memory resource amount occupied by the target data is released, the target data set includes data that has been loaded into the memory and is not in the priority data queue, and / or data that has been loaded into the memory and has a priority lower than the priority of the first data;

[0220] When the data volume of the above-mentioned first data is smaller than the current amount of free resources, load the above-mentioned first data; when the data volume of the above-mentioned first data is not smaller than the current amount of free resources, return to step: select the target data in the target data set and release the memory resources occupied by the above-mentioned target data to continue execution until the data volume of the above-mentioned first data is smaller than the current amount of free resources, or all the data in the above-mentioned target data set have been released.

[0221] In an optional implementation, the release module is specifically used for:

[0222] The data that has been loaded into the memory and is not in the priority data queue is preferentially selected as the target data.

[0223] In an optional implementation, the release module is specifically used for:

[0224] The data that has been loaded into the memory and is not in the priority data queue is taken as target data in order from large to small in terms of the amount of memory resources occupied.

[0225] In an optional implementation manner, the historical query information includes: query time and data loading time, and the determination module is specifically used to:

[0226] Prioritize your data according to the following formula;

[0227] initial_order(i)

[0228] =query_time_order(i)×c3(i)+query_frequency_order(i)×c4(i)+load_cost_order(i)×c4(i)

[0229] Among them, query_time_order(i) is the ranking of data i in the entire data set in reverse order of query time; query_frequency_order is the ranking of data i in the entire data set in ascending order of query frequency; load_cost_order(i) is the ranking of data i in the entire data set in reverse order of load time; c3, c4, c5 are preset weight values, and initial_order(i) is the priority of data i.

[0230] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0231] The present application also provides an electronic device, such as Figure 4 As shown, including:

[0232] Memory 401, used for storing computer programs;

[0233] The processor 402 is used to execute the program stored in the memory 401 to implement the following steps:

[0234] Obtaining historical query information of data of each data set; each of the above data sets includes at least one piece of data;

[0235] According to the above historical query information, the priority of each of the above data is determined respectively;

[0236] For each data set, select the data with the highest priority from the data that have not been selected in the data set;

[0237] Obtain the current free resource amount of the memory, and if the current free resource amount is not less than the data amount of each currently selected data, load each currently selected data;

[0238] Return to the execution step: for each data set, select the data with the highest priority from the data that has not been selected in the data set, until the current free resource amount is insufficient or the data in each of the data sets are loaded.

[0239] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 402, the communication interface, and the memory 401 communicate with each other via the communication bus.

[0240] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0241] The communication interface is used for communication between the above electronic device and other devices.

[0242] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0243] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0244] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data loading methods are implemented.

[0245] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the data loading methods in the above embodiments.

[0246] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0247] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0248] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0249] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A data loading method, characterized in that: The method comprises: Acquire historical query information of data of each data set; each of the data sets includes at least one piece of data; Determining the priority of each of the data respectively according to the historical query information; For each data set, select the data with the highest priority from the data that have not been selected in the data set; Acquire the current free resource amount of the memory, and load the currently selected data if the current free resource amount is not less than the data amount of the currently selected data; Return to the execution step: for each data set, select the data with the highest priority from the data that has not been selected in the data set, until the current free resource amount is insufficient or the data in each of the data sets are all loaded.

2. The method according to claim 1, characterized in that The data includes subsystem information and environment information, wherein the subsystem information represents the source of the data, and the environment information represents the application scenario of the data; The method further comprises: Determine the subsystem grouping conditions and environment grouping conditions; According to the subsystem grouping condition and the environment grouping condition, each data is grouped based on the subsystem information and environment information of each data to obtain each data set.

3. The method according to claim 1, characterized in that The historical query information includes at least one of the following parameters: query frequency, query time, and data loading time, wherein the higher the query frequency of the data, the higher the priority of the data; the shorter the time from the last query time to the current time, the higher the priority of the data; and the shorter the data loading time, the higher the priority of the data.

4. The method according to claim 3, characterized in that The method further comprises: Based on the query time, determine an average query interval time for the data; Determining the priority of each data according to the historical query information includes: Determining an initial priority of data according to the historical query information; The priority of the data is determined according to the initial priority of the data and the average query interval time of the data.

5. The method according to claim 4, characterized in that The determining, based on the query time, an average query interval time of the data includes: After each query is completed, the time interval between two adjacent queries is recorded in real time, and the query interval time is calculated based on each time interval; The current average query interval is calculated according to the following formula: avg_query_interval'=avg_query_interval*c2 avg_query_interval is the query interval, c2 is the preset threshold coefficient, and avg_query_interval' is the average query interval; and / or In the case that the current free resource amount is not less than the data amount of each currently selected data, after loading each currently selected data, the method further includes: Calculate the current memory availability information according to the following formula: free_mem=total_mem*c1–used_mem Where free_mem is the current memory free information, total_mem is the total system memory, used_mem is the memory information occupied by the system, and c1 is the preset threshold coefficient.

6. The method according to claim 1, characterized in that The obtaining of the current free resource amount of the memory, and loading the currently selected data when the current free resource amount is not less than the data amount of the currently selected data, comprises: Sort the currently selected data in descending order of priority to obtain a priority data queue; Selecting the data at the head of the queue in the priority data queue to obtain first data, wherein the selected first data is removed from the priority data queue; Acquire a current free resource amount of the memory, and load the first data if the current free resource amount is not less than the data amount of the first data; Return to the execution step: select the data at the head of the queue in the priority data queue to obtain the first data, until all the currently selected data are loaded.

7. The method according to claim 6, characterized in that The obtaining of the current free resource amount of the memory, and loading the first data when the current free resource amount is not less than the data amount of the first data, includes: Acquire the current free resource amount of the memory, and if the current free resource amount is not less than the data amount of the first data, compare the first data with the data loaded in the memory, and if the comparison is unsuccessful, load the first data; If the comparison is successful, it is determined that the first data has been loaded into the memory.

8. The method according to claim 6, characterized in that The method further comprises: In a case where the current free resource amount is less than the data amount of the first data, selecting target data from a target data set and releasing the memory resource amount occupied by the target data, the target data set including data that has been loaded into the memory and is not in the priority data queue, and / or data that has been loaded into the memory and has a priority lower than that of the first data; When the data volume of the first data is smaller than the current free resource volume, load the first data; when the data volume of the first data is not smaller than the current free resource volume, return to step: select the target data in the target data set and release the memory resources occupied by the target data set to continue execution until the data volume of the first data is smaller than the current free resource volume, or all the data in the target data set has been released.

9. The method according to claim 8, characterized in that The step of selecting target data from the target data set and releasing the memory resources occupied by the target data includes: The data that has been loaded into the memory and is not in the priority data queue is taken as target data in order from large to small in terms of the amount of memory resources occupied.

10. The method according to claim 1, characterized in that The historical query information includes: query time, data loading time, and for each piece of data, determining the priority of the data according to the historical query information of the data includes: Prioritize your data based on the following formula; initial_order(i)=query_time_order(i)×c3(i)+query_frequency_order(i)×c4(i)+load_cost_order(i)×c4(i) Among them, query_time_order(i) is the ranking of data i in the entire data set in reverse order of query time; query_frequency_order is the ranking of data i in the entire data set in ascending order of query frequency; load_cost_order(i) is the ranking of data i in the entire data set in reverse order of load time; c3, c4, c5 are preset weight values, and initial_order(i) is the priority of data i.

11. A data loading device, characterized in that: The device comprises: An acquisition module, used to acquire historical query information of data of each data set; each of the data sets includes at least one piece of data; A determination module, used to determine the priority of each data according to the historical query information; A selection module, for selecting, for each data set, data with the highest priority from data that has not been selected in the data set; A loading module, used for obtaining the current free resource amount of the memory, and loading each currently selected data when the current free resource amount is not less than the data amount of each currently selected data; The processing module is used to return to the execution step: for each data set, the data with the highest priority is selected from the data that has not been selected in the data set, until the current free resource amount is insufficient or the data in each of the data sets are loaded.

12. The device according to claim 11, characterized in that The historical query information includes at least one of the following parameters: query frequency, query time, and data loading time, wherein the higher the query frequency of data, the higher the priority of the data; the shorter the time from the last query time to the current time, the higher the priority of the data; and the shorter the data loading time, the higher the priority of the data; The data includes subsystem information and environment information, wherein the subsystem information represents the source of the data, and the environment information represents the application scenario of the data; The device also includes: A grouping determination module, used to determine subsystem grouping conditions and environment grouping conditions; A grouping module, used to group each data according to the subsystem grouping condition and the environment grouping condition, based on the subsystem information and environment information of each data, to obtain each data set; The device also includes: An average query interval time determination module, used to determine an average query interval time of data based on the query time; The determination module is specifically used for: Determining an initial priority of data according to the historical query information; Determine the priority of the data according to the initial priority of the data and the average query interval time of the data; The average query interval time determination module is specifically used for: After each query is completed, the time interval between two adjacent queries is recorded in real time, and the query interval time is calculated based on each time interval; The current average query interval is calculated according to the following formula: avg_query_interval'=avg_query_interval*c2 avg_query_interval is the query interval, c2 is the preset threshold coefficient, and avg_query_interval' is the average query interval; and / or The device also includes: The memory free space calculation module is used to calculate the current memory free space information according to the following formula after loading the currently selected data when the current free resource amount is not less than the data amount of each currently selected data: free_mem=total_mem*c1–used_mem Where free_mem is the current memory free information, total_mem is the total system memory, used_mem is the memory information occupied by the system, and c1 is the preset threshold coefficient; The loading module is specifically used for: Sort the currently selected data in descending order of priority to obtain a priority data queue; Selecting the data at the head of the queue in the priority data queue to obtain first data, wherein the selected first data is removed from the priority data queue; Acquire a current free resource amount of the memory, and load the first data if the current free resource amount is not less than the data amount of the first data; Return to the execution step: select the data at the head of the queue in the priority data queue to obtain the first data, until all the currently selected data are loaded; The loading module is specifically used for: Acquire the current free resource amount of the memory, and if the current free resource amount is not less than the data amount of the first data, compare the first data with the data loaded in the memory, and if the comparison is unsuccessful, load the first data; If the comparison is successful, determining that the first data has been loaded into the memory; The device also includes: Release module for: In a case where the current free resource amount is less than the data amount of the first data, selecting target data from a target data set and releasing the memory resource amount occupied by the target data, the target data set including data that has been loaded into the memory and is not in the priority data queue, and / or data that has been loaded into the memory and has a priority lower than that of the first data; If the data volume of the first data is less than the current free resource volume, the first data is loaded; if the data volume of the first data is not less than the current free resource volume, the process returns to step: selecting target data in the target data set and releasing the memory resource volume occupied by the target data set to continue the process until the data volume of the first data is less than the current free resource volume, or all the data in the target data set has been released; The release module is specifically used for: Preferentially selecting the data that has been loaded into the memory and is not in the priority data queue as the target data; The release module is specifically used for: In descending order of the amount of memory resources occupied, the data that has been loaded into the memory and is not in the priority data queue is used as target data; The historical query information includes: query time, data loading time, and the determination module is specifically used for: Prioritize your data based on the following formula; initial_order(i)=query_time_order(i)×c3(i)+query_frequency_order(i)×c4(i)+load_cost_order(i)×c4(i) Among them, query_time_order(i) is the ranking of data i in the entire data set in reverse order of query time; query_frequency_order is the ranking of data i in the entire data set in ascending order of query frequency; load_cost_order(i) is the ranking of data i in the entire data set in reverse order of load time; c3, c4, c5 are preset weight values, and initial_order(i) is the priority of data i.

13. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-10 when executing a program stored in a memory.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images