Data loading method and device, storage medium and electronic device

Abstract

Embodiments of the present application provide a data loading method and device, a storage medium and an electronic device, wherein the method comprises: in the case of receiving a starting instruction, performing a first loading operation on first target data stored in a target memory according to a first preset rule, the starting instruction being used to indicate loading target data into the target memory, the target data comprising the first target data; in the case of determining that the first loading operation is completed, performing a second loading operation on second target data based on the current remaining capacity of the target memory, the second target data comprising incremental data of data currently stored in the target memory relative to data stored in the target memory before performing the first loading operation. Through the present application, the problem that data is large and is likely to cause memory overflow and thus program performance to decline in the related art is solved, and the effect of improving program performance stability is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of big data technology, and more specifically, to a data loading method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the advent of the information age, the scale of data is growing exponentially, leading to a stronger reliance on memory and increasingly larger cluster sizes. Spark, as a big data processing engine, is a popular computing framework in the field of big data applications. A key characteristic of most big data application scenarios, such as image search, vehicle search, and peer-to-peer (PAP) analysis, is the strong correlation between time and data value; that is, newer data has higher value. However, changes in pedestrian and vehicle traffic, as well as adjustments in the number of surveillance cameras, can easily cause memory overflow issues, leading to performance degradation, inaccurate task results, and even functional abnormalities. The stability and performance of programs are severely tested in the face of various unforeseen circumstances. In short, related technologies suffer from memory overflow and performance degradation issues when dealing with large amounts of data.

[0003] There is currently no effective solution to the problem that large amounts of data can easily cause memory overflows, leading to a decrease in program performance. Summary of the Invention

[0004] This invention provides a data loading method, apparatus, storage medium, and electronic device to at least solve the problem in related technologies where large amounts of data can easily cause memory overflow, leading to a decrease in program performance.

[0005] According to an embodiment of the present invention, a data loading method is provided, comprising: upon receiving a start instruction, performing a first loading operation on first target data stored in a target memory according to a first preset rule, wherein the start instruction is used to instruct the target data to be loaded into the target memory, the target data including the first target data; and upon determining that the first loading operation has been completed, performing a second loading operation on second target data based on the current remaining capacity of the target memory, wherein the second target data includes incremental data of the data currently stored in the target memory relative to the data stored in the target memory before the first loading operation was performed.

[0006] In an exemplary embodiment, performing a second loading operation on second target data based on the current remaining capacity of the target memory includes: determining the target capacity required by the second target data; determining whether the target capacity and the current remaining capacity of the target memory satisfy a first preset condition; and performing the second loading operation on the second target data if it is determined that the target capacity and the current remaining capacity of the target memory satisfy the first preset condition.

[0007] In an exemplary embodiment, after determining whether the target capacity and the current remaining capacity of the target memory satisfy a first preset condition, the method further includes: if it is determined that the target capacity and the current remaining capacity of the target memory do not satisfy the first preset condition, continuously cleaning the target memory in a preset manner; if it is determined that the target capacity and the remaining capacity after cleaning the target memory satisfy the first preset condition, performing the second loading operation on the second target data.

[0008] In one exemplary embodiment, continuously cleaning the target memory in a preset manner includes at least one of the following: continuously cleaning the target memory at a predetermined time span granularity; or continuously cleaning the target memory in a predetermined time sequence.

[0009] In an exemplary embodiment, performing a second loading operation on the second target data based on the current remaining capacity of the target memory includes: performing a second loading operation on the second target data according to a preset period based on the current remaining capacity of the target memory.

[0010] In an exemplary embodiment, after performing a first loading operation on first target data stored in a target memory according to a first preset rule, the method further includes: recording first time span information for performing the first loading operation on the first target data, wherein the cutoff time included in the first time span information is the time when the first loading operation begins; after performing a second loading operation on second target data based on the current remaining capacity of the target memory, the method further includes: recording second time span information for performing the second loading operation on the second target data, wherein the start time included in the second time span information is the cutoff time included in the first time span information.

[0011] In an exemplary embodiment, performing a first loading operation on first target data stored in a target memory according to a first preset rule includes: obtaining time span configuration information, wherein the time span configuration information is used to indicate the time span information for generating the target data; determining the time span granularity of the first loading operation based on the time span configuration information; dividing the first target data into one or more sub-target data according to the time span granularity; and performing the first loading operation on one or more sub-target data in a backward-to-forward order, using the time span granularity as the unit, to load the first target data into the target memory.

[0012] In an exemplary embodiment, the method further includes performing the following operations for each of the sub-target data included in the first target data: estimating the sub-target capacity of the sub-target data before loading the sub-target data; determining whether the sub-target capacity and the current remaining capacity of the target memory satisfy a second preset condition; if it is determined that the sub-target capacity and the current remaining capacity of the target memory satisfy the second preset condition, loading the sub-target data into the target memory; if it is determined that the sub-target capacity and the current remaining capacity of the target memory do not satisfy the second preset condition, stopping the first loading operation and determining the data already loaded into the target memory as the first target data.

[0013] In one exemplary embodiment, after performing a second loading operation on the second target data based on the target memory capacity, the method further includes: merging multiple resilient distributed datasets (RDDs) included in the target memory, wherein the second target data includes multiple RDDs.

[0014] In an exemplary embodiment, after performing a second loading operation on the second target data based on the current remaining capacity of the target memory, the method further includes: if it is determined that the data in the target memory has been updated, loading the updated data in the target memory with the same time span as the time span of the first target data into the target memory according to a second preset rule.

[0015] In an exemplary embodiment, after performing a second loading operation on the second target data based on the current remaining capacity of the target memory, the method further includes: upon receiving a target instruction, performing the first loading operation on the third target data stored in the target memory according to the first preset rule, wherein the target instruction is an instruction triggered after resetting the time span configuration information.

[0016] According to another embodiment of the present invention, a data loading apparatus is also provided, comprising: a first loading module, configured to perform a first loading operation on first target data stored in a target memory according to a first preset rule upon receiving a start instruction, wherein the start instruction is used to instruct the target data to be loaded into the target memory, the target data including the first target data; and a second loading module, configured to perform a second loading operation on second target data based on the currently remaining capacity of the target memory upon determining that the first loading operation has been completed, wherein the second target data includes incremental data of the data currently stored in the target memory relative to the data stored in the target memory before the execution of the first loading operation.

[0017] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0018] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0019] This invention achieves the goal of avoiding the risk of memory overflow by loading data based on the remaining capacity of memory space. This is achieved by performing a first loading operation on the first target data stored in the target memory according to a first preset rule, and then performing a second loading operation on the incremental data generated after the first loading operation started, based on the remaining capacity of the target memory. Therefore, it solves the problem in related technologies where large amounts of data can easily cause memory overflow, leading to a decrease in program performance, and thus improves the stability of program performance. Attached Figure Description

[0020] Figure 1 This is a block diagram of the mobile terminal hardware structure of the data loading method according to an embodiment of the present invention;

[0021] Figure 2 This is a flowchart of a data loading method according to an embodiment of the present invention;

[0022] Figure 3 This is a flowchart of a data loading method according to a specific embodiment of the present invention;

[0023] Figure 4 This is a structural block diagram of a data loading device according to an embodiment of the present invention. Detailed Implementation

[0024] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0026] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a block diagram of the mobile terminal hardware structure of the data loading method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0027] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data loading method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0029] This embodiment provides a data loading method. Figure 2 This is a flowchart of a data loading method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0030] Step S202: Upon receiving a startup command, a first loading operation is performed on the first target data stored in the target memory according to a first preset rule, wherein the startup command is used to instruct the target data to be loaded into the target memory, and the target data includes the first target data;

[0031] Step S204: If the first loading operation is completed, a second loading operation is performed on the second target data based on the current remaining capacity of the target memory, wherein the second target data includes incremental data of the data currently stored in the target memory relative to the data stored in the target memory before the first loading operation was performed.

[0032] Through the above steps, a first loading operation is performed on the first target data stored in the target memory according to a first preset rule. After the first loading operation is completed, a second loading operation is performed on the incremental data generated in the target memory after the first loading operation began, based on the current remaining capacity of the target memory. This achieves the goal of loading data based on the remaining memory space to avoid the risk of memory overflow. Therefore, it solves the problem in related technologies where large amounts of data can easily cause memory overflow, leading to a decrease in program performance, and achieves the effect of improving program performance stability.

[0033] The entity executing the above steps can be a terminal, a server, or a backend processing system, such as a backend processing device, a backend management device, or a backend processing module, or a processor with human-computer interaction capabilities configured on a storage device, or a processing device or processing unit with similar processing capabilities, but is not limited to these. The following explanation uses the server executing the above operations as an example (this is merely an illustrative example; in actual operation, other devices or modules can also perform the above operations):

[0034] In the above embodiments, upon receiving a startup command, the server performs a first loading operation on the first target data stored in the target memory according to a first preset rule. Taking a big data application scenario as an example, for instance, a user might want to compare or calculate data stored in the target memory for a predetermined time span (such as 7 days, 30 days, or other time spans). For example, the first target data could be audio, video, or image data collected by a front-end camera device. The startup command could be an instruction triggered by the user performing an operation on a terminal, APP, or Web. Upon receiving the startup command, the server loads the first target data into the target memory, which could be Spark memory or other memory. That is, after receiving the startup command, the server loads the first target data (such as data from the 7 days prior to receiving the startup command). Once the first loading operation is completed, the server loads the first target data (such as data from the 7 days prior to receiving the startup command). A second loading operation is performed on the second target data using the remaining capacity of the target memory. The second target data includes the incremental data generated after the first loading operation was started and stored in the current target memory. That is, the second loading operation is performed on the newly generated incremental data to load the newly generated incremental data into the target memory (such as the Spark memory mentioned above). Based on the completion of loading the first target data, the latest data can also be loaded. This embodiment is applicable to big data application scenarios, especially big data scenarios with strong time correlation. It realizes the purpose of dynamically caching big data based on the remaining capacity of memory, and further realizes the purpose of processing the cached big data. It avoids the problem of memory overflow caused by large data in related technologies, which leads to a decrease in program performance, and achieves the effect of improving program performance stability.

[0035] In an optional embodiment, performing a second loading operation on the second target data based on the current remaining capacity of the target memory includes: determining the target capacity required by the second target data; determining whether the target capacity and the current remaining capacity of the target memory satisfy a first preset condition; and performing the second loading operation on the second target data if the target capacity and the current remaining capacity of the target memory satisfy the first preset condition. In this embodiment, when performing the second loading operation on the second target data, the target capacity occupied by the second target data is first determined, that is, the memory resources occupied by the second target data are estimated. Then, it is determined whether the target capacity and the current remaining capacity of the target memory (such as the Spark memory mentioned above) satisfy the first preset condition. For example, the first preset condition is that the difference between the current remaining capacity of the target memory and the target capacity (i.e., the remaining capacity after loading) is not lower than a preset threshold. For example, the preset threshold is a predetermined proportion of the maximum capacity of the target memory (such as 10%, 5%, or other values). If the target capacity and the current remaining capacity of the target memory satisfy the first preset condition, the second loading operation is performed on the second target data. This embodiment achieves the goal of loading incremental data based on the current remaining capacity of the target memory, ensuring that memory overflow will not cause a decrease in program performance, while also achieving the goal of loading the latest data.

[0036] In an optional embodiment, after determining whether the target capacity and the current remaining capacity of the target memory meet a first preset condition, the method further includes: if it is determined that the target capacity and the current remaining capacity of the target memory do not meet the first preset condition, continuously cleaning the target memory according to a preset method; if it is determined that the target capacity and the remaining capacity after cleaning the target memory meet the first preset condition, performing the second loading operation on the second target data. In this embodiment, when it is determined that the target capacity and the current remaining capacity of the target memory do not meet the first preset condition, the target memory (such as the Spark memory mentioned above) is cleaned according to a preset method until it is determined that the target capacity and the remaining capacity after cleaning the target memory meet the first preset condition. Only then is the second loading operation performed on the second target data. That is, when the remaining capacity of the target memory is insufficient, the target memory is cleaned to free up enough memory space to load the latest data. For example, the oldest data can be cleaned first in chronological order until memory is sufficient. Alternatively, the target memory can be cleaned at a time granularity, such as by day or month. Optionally, in practical applications, data can be cleaned by day and in chronological order from early to late (or from old to new) until memory is sufficient. This embodiment achieves the purpose of cleaning up memory when memory space is insufficient, thereby caching large amounts of data with strong correlation to time.

[0037] In an optional embodiment, continuously cleaning the target memory according to a preset method includes at least one of the following: continuously cleaning the target memory according to a predetermined time span granularity; continuously cleaning the target memory according to a predetermined time sequence. In this embodiment, the target memory can be cleaned according to the time span granularity, for example, by day or month, i.e., cleaning a whole day's worth of data or a whole month's worth of data until memory is sufficient; the target memory can also be cleaned according to the time sequence, for example, cleaning data in the target memory in order from morning to night (or from oldest to newest), i.e., cleaning the oldest data and retaining relatively newer data; optionally, in practical applications, when the remaining capacity of the target memory is insufficient, the target memory can be cleaned simultaneously according to both the time span granularity and the time sequence. This embodiment achieves flexible cleaning of the target memory to store newly added data.

[0038] In an optional embodiment, performing a second loading operation on the second target data based on the current remaining capacity of the target memory includes: performing a second loading operation on the second target data according to a preset period based on the current remaining capacity of the target memory. In this embodiment, the second loading operation on the second target data can be performed according to a predetermined period, for example, periodically performing the second loading operation on the second target data (or incremental data, or newly added data) every 5 minutes, 10 minutes, or other time intervals, so as to achieve the purpose of loading the latest data into the target memory (such as the Spark memory mentioned above) in a timely manner, thereby enabling the comparison or calculation of the latest data and historical data. For application scenarios where time and data value are strongly correlated, the newer the data, the higher its value. The technical solution corresponding to this embodiment can achieve the purpose of maximizing the satisfaction of user needs.

[0039] In an optional embodiment, after performing a first loading operation on the first target data stored in the target memory according to a first preset rule, the method further includes: recording first time span information for performing the first loading operation on the first target data, wherein the cutoff time included in the first time span information is the time when the first loading operation begins; after performing a second loading operation on the second target data based on the current remaining capacity of the target memory, the method further includes: recording second time span information for performing the second loading operation on the second target data, wherein the start time included in the second time span information is the cutoff time included in the first time span information. In this embodiment, after performing the first loading operation on the first target data, the first time span information of performing the first loading operation on the first target data is recorded. In practical applications, when the server receives a startup command, for example, a startup command triggered by a user performing an operation on a client, APP, or Web client, the startup command includes the user-set time span configuration information of the data to be loaded into memory (such as 7 days, 30 days, or other time spans). This configuration indicates the time span of data to be loaded backward from the current time. That is, the end time in the above-mentioned first time span information is the time when the startup command is received and the first loading operation begins, while the start time in the first time span information is the time indicated in the time span configuration information backward from the end time (such as 7 days as mentioned above). Similarly, after performing the second loading operation on the second target data, the second time span information of performing the second loading operation on the second target data is also recorded, and the start time in the second time span information is the end time in the above-mentioned first time span information. Through this embodiment, the purpose of recording the data loading time span information after performing the first loading operation and the second loading operation is achieved.

[0040] In an optional embodiment, performing a first loading operation on the first target data stored in the target memory according to a first preset rule includes: obtaining time span configuration information, wherein the time span configuration information is used to indicate the time span information for generating the target data; determining the time span granularity of the first loading operation based on the time span configuration information; dividing the first target data into one or more sub-target data according to the time span granularity; and performing the first loading operation on one or more sub-target data in a backward-to-forward order, using the time span granularity as the unit, to load the first target data into the target memory. In this embodiment, performing the first loading operation on the first target data according to the first preset rule includes: determining the time span granularity based on the time span configuration information. For example, if the time span configuration information is set by the user to load data for 7 days, 30 days, or other numbers of days, then the time span granularity can be determined to be 1 day. Of course, in practical applications, if the time span indicated by the time span configuration is longer, such as requiring the loading of data for 1 year or half a year, then the time span granularity can be determined to be 1 month. Then, according to the time span granularity (such as 1 day as mentioned above), the first target data (for example, data with a time span configured to be 7 days) is divided into multiple sub-target data (i.e., 7 sub-target data), and then loaded cyclically in a backward order (i.e., in reverse order). The system loads seven sub-target data sets, prioritizing the data most recent to the current time and loading it into the target memory. Then, it loads data from other days sequentially, day by day, until all data is loaded. In practical applications, if the target memory reaches a preset threshold before all data is loaded (a predetermined percentage of the target memory's maximum capacity—for example, after loading six days of data, if the remaining target memory capacity is less than a predetermined percentage of its maximum capacity (e.g., 10%, 5%, or other values), it ensures that newer data is loaded into the target memory to reflect the application objective that newer data has higher value). Alternatively, in practical applications, memory can be cleared to ensure all the first target data is loaded into the target memory. This embodiment achieves the goal of loading the first target data into the target memory based on time span configuration information.

[0041] In an optional embodiment, the method further includes: for each of the sub-target data included in the first target data, performing the following operations: before loading the sub-target data, estimating the sub-target capacity of the sub-target data; determining whether the sub-target capacity and the current remaining capacity of the target memory satisfy a second preset condition; if it is determined that the sub-target capacity and the current remaining capacity of the target memory satisfy the second preset condition, loading the sub-target data into the target memory; if it is determined that the sub-target capacity and the current remaining capacity of the target memory do not satisfy the second preset condition, stopping the first loading operation, and determining the data already loaded into the target memory as the first target data. In this embodiment, a loading operation is performed on each sub-target data (such as the aforementioned 7 sub-target data) included in the first target data based on the remaining capacity of the target memory. The specific operation is as follows: First, the sub-target capacity of the sub-target data (or the data block with the smallest time granularity) is determined, that is, the memory resources required by the sub-target data are estimated. Then, it is determined whether the sub-target capacity and the current remaining capacity of the target memory (such as the aforementioned Spark memory) meet a second preset condition. For example, the second preset condition is that the difference between the current remaining capacity of the target memory and the target capacity (i.e., the remaining capacity after loading) is not lower than a preset threshold. For example, the preset threshold is a predetermined proportion of the maximum capacity of the target memory (such as 10%, or 5%, or other values). If the sub-target capacity and the current remaining capacity of the target memory meet the second preset condition, the sub-target data is loaded into the target memory. If the sub-target capacity and the current remaining capacity of the target memory do not meet the second preset condition, the loading operation is stopped. This embodiment achieves the goal of loading each sub-target data included in the first target data sequentially based on the current remaining capacity of the target memory, avoiding the problem in related technologies where large amounts of data can easily cause memory overflow, leading to a decrease in program performance.

[0042] In an optional embodiment, after performing a second loading operation on the second target data based on the target memory capacity, the method further includes: merging multiple Resilient Distributed Datasets (RDDs) included in the target memory, wherein the second target data includes multiple RDDs. In this embodiment, multiple RDDs included in the target memory can be merged. In practical applications, multiple RDDs can be merged periodically, for example, every 5 hours or 3 hours, the RDDs storing data in memory can be merged. The RDDs are incremental data generated after the first loading operation begins. When the second loading operation is performed on the incremental data according to a predetermined period (e.g., 5 minutes), many RDDs will be generated. Through this embodiment, the number of RDDs can be reduced to improve retrieval performance.

[0043] In an optional embodiment, after performing a second loading operation on the second target data based on the current remaining capacity of the target memory, the method further includes: if it is determined that the data in the target memory has been updated, loading the updated data in the target memory with the same time span as the first target data into the target memory according to a second preset rule. In this embodiment, when the stored data has been modified or deleted, the updated data can be loaded into the target memory. For example, a full load can be performed (corresponding to the aforementioned first loading operation) to load the updated data with the same time span as the aforementioned first target data (such as the aforementioned 7-day data) into the target memory. In practical applications, a full load can also be performed periodically during periods of low business activity to update the data in the target memory, or a full load can be performed actively to update the data in the target memory. Through this embodiment, the purpose of performing a full load on the updated data to update the cached data in the target memory when the data has been updated is achieved.

[0044] In an optional embodiment, after performing a second loading operation on the second target data based on the current remaining capacity of the target memory, the method further includes: upon receiving a target instruction, performing the first loading operation on the third target data stored in the target memory according to the first preset rule, wherein the target instruction is an instruction triggered after resetting the time span configuration information. In this embodiment, upon receiving the target instruction, for example, if the target instruction is a user-reset data loading time span configuration, such as the original time span configuration being 7 days and now being changed to 21 days, the first loading operation (e.g., a full load) can be actively triggered to load the third target data stored in the target memory into the target memory. In practical applications, at this time, the incremental loading process (as described in the aforementioned second loading operation) will be blocked, and the loading operation on the third target data will be performed. For example, data from the 8th to the 21st day prior to the current time will be loaded into the target memory through a full load. After the full load is completed, the incremental data will be loaded into the target memory.

[0045] Obviously, the embodiments described above are only some embodiments of the present invention, and not all embodiments. The present invention will be specifically described below with reference to the embodiments.

[0046] This invention takes into account the big data scenario, where the continuous growth of data volume over time is a crucial aspect. Newer data has higher usage value, and this invention's embodiments are applicable to big data scenarios with a strong time dimension correlation. The amount of data loaded is determined based on allocated memory resources, including full and incremental processes.

[0047] Figure 3 This is a flowchart of a data loading method according to a specific embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0048] S302, the program starts and reads the time span configuration that needs to be loaded into memory. This configuration represents the time span of the data that needs to be loaded, counting backward from the current time.

[0049] S304: Obtain the minimum time span granularity (e.g., 1 day) for full loading (corresponding to the first loading operation mentioned above). During the first full loading of data into memory (corresponding to the target memory mentioned above, or the Spark memory mentioned above), estimate the memory resources occupied by the data. If there is enough memory, load the data blocks of the minimum time granularity (corresponding to the sub-target data mentioned above) in reverse time order, that is, perform full loading in segments and update the remaining memory size record.

[0050] S306, If the data loading is complete, then stop loading;

[0051] S308: If memory usage reaches the threshold or there is insufficient memory, loading will stop.

[0052] S310 records the data loading time span, updates the remaining memory size record, and the program startup is complete;

[0053] It should be noted that the above steps S306 and S308 are not in any particular order. If any one of the conditions is met, the loading will end and the program will proceed to step S310, at which point the program will start up.

[0054] S312, Incremental loading (corresponding to the aforementioned second loading operation) is timed loading (e.g., 5 minutes), that is, the incremental loading process is triggered by timed timing, and execution begins after the above program has started;

[0055] S314, estimate the memory resources occupied by the data, and if the remaining memory (corresponding to the current remaining capacity of the aforementioned target memory) is insufficient, cyclically clean up the data with the earliest and smallest time granularity (e.g., 1 day) until enough memory space is freed up, then load that part of the data, that is, clean up a sufficient amount of the oldest data, and continue loading until the incremental loading is completed;

[0056] S316: When there is sufficient memory, it loads directly until incremental loading is complete, updates the recorded data loading time span, and updates the remaining memory size record.

[0057] After incremental loading is complete, the S318 can perform in-memory computing tasks.

[0058] Furthermore, after completing the above steps, the RDDs storing data in memory can be merged periodically (e.g., every 5 hours) to update the remaining memory size record. Reducing the number of RDDs can improve retrieval performance.

[0059] Furthermore, after completing the above steps, if there are any data modification or deletion operations, a full load can be performed periodically (e.g., once a day) during periods with less business to update the cached data.

[0060] Furthermore, after completing the above steps, a full load can be actively triggered. This mainly addresses the issue of resetting the data loading time span, blocking the incremental loading process, and performing a full load without restarting the program. After the full load process is completed, the incremental data blocked during the full load process is loaded, and then the incremental process is started.

[0061] Through the embodiments of the present invention, the data to be loaded is determined based on the remaining memory, which ensures both full utilization of resources and stable operation of the program. By executing the full process and the incremental process respectively, the newly added data can participate in memory calculation within a minute time scale. In scenarios where newer data has higher value, the validity of the results is not reduced at all, while the program performance can be effectively improved.

[0062] In the above embodiments, based on time-sensitive big data scenarios, while prioritizing functionality (by loading data from memory space to avoid the risk of memory overflow), performance is not compromised (pure memory computing). Based on time-dimensional priority settings, intelligent dynamic memory management is achieved, which can effectively improve system stability and meet user needs to the greatest extent possible with existing resources.

[0063] Compared with existing technologies, the embodiments of the present invention achieve the following effects: users do not need to worry about service anomalies caused by increased data volume; the server's cache resource usage is monitored in real time, ensuring full utilization of resources while ensuring normal program operation; full data loading is performed in chronological order, ensuring the latest data is in memory when memory is insufficient to load all data, and the present invention assumes that newer data has greater value, providing maximum data value while ensuring program functionality; in the event of a sudden increase in data, memory resources can be fully utilized while ensuring normal program operation, and program stability should not presuppose user awareness of increased camera coverage, surges in pedestrian and vehicle traffic, etc., requiring manual adjustment of the data loading time span, the present invention increases program stability in handling various scenarios; in the event of incremental data loading and insufficient memory, the oldest data is deleted, and data is loaded after freeing up sufficient memory space, providing maximum data value while ensuring program functionality; in scenarios where newer data has higher value, only data already loaded into memory is selected for calculation (without compression or disk access), returning more valuable results based on more valuable data and improving program performance.

[0064] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0065] This embodiment also provides a data loading device. Figure 4 This is a structural block diagram of a data loading device according to an embodiment of the present invention, such as... Figure 4 As shown, the device includes:

[0066] The first loading module 402 is configured to perform a first loading operation on the first target data stored in the target memory according to a first preset rule when a startup instruction is received, wherein the startup instruction is used to instruct the target data to be loaded into the target memory, and the target data includes the first target data;

[0067] The second loading module 404 is configured to perform a second loading operation on the second target data based on the current remaining capacity of the target memory when the first loading operation is determined to be completed, wherein the second target data includes incremental data of the data currently stored in the target memory relative to the data stored in the target memory before the first loading operation was performed.

[0068] In an optional embodiment, the second loading module 404 includes: a first determining unit, configured to determine the target capacity required by the second target data; a first judging unit, configured to judge whether the target capacity and the current remaining capacity of the target memory satisfy a first preset condition; and a first executing unit, configured to perform the second loading operation on the second target data when it is determined that the target capacity and the current remaining capacity of the target memory satisfy the first preset condition.

[0069] In an optional embodiment, the above apparatus further includes: a cleaning module, configured to continuously clean the target memory according to a preset method after determining whether a first preset condition is met between the target capacity and the current remaining capacity of the target memory, and when it is determined that the target capacity and the current remaining capacity of the target memory do not meet the first preset condition; and a first execution module, configured to perform the second loading operation on the second target data when it is determined that the target capacity and the remaining capacity after cleaning the target memory meet the first preset condition.

[0070] In an optional embodiment, the cleanup module can clean up the target memory in at least one of the following ways: continuously clean up the target memory at a predetermined time span granularity; or continuously clean up the target memory in a predetermined time sequence.

[0071] In an optional embodiment, the second loading module 404 includes a second execution unit, configured to perform a second loading operation on the second target data according to a preset period based on the current remaining capacity of the target memory.

[0072] In an optional embodiment, the above apparatus further includes: a first recording module, configured to record first time span information of performing the first loading operation on the first target data after performing a first loading operation on the first target data stored in the target memory according to a first preset rule, wherein the cutoff time included in the first time span information is the time when the first loading operation is started; and a second recording module, configured to record second time span information of performing the second loading operation on the second target data after performing a second loading operation on the second target data based on the current remaining capacity of the target memory, wherein the start time included in the second time span information is the cutoff time included in the first time span information.

[0073] In an optional embodiment, the first loading module 402 includes: an acquisition unit, configured to acquire time span configuration information, wherein the time span configuration information is used to indicate the time span information for generating the target data; a second determination unit, configured to determine the time span granularity of the first loading operation based on the time span configuration information; a division unit, configured to divide the first target data into one or more sub-target data according to the time span granularity; and a third execution unit, configured to execute the first loading operation on one or more sub-target data in a backward-to-forward order, based on the time span granularity, to load the first target data into the target memory.

[0074] In an optional embodiment, the above apparatus further includes: a second execution module, configured to perform the following operations for each of the sub-target data included in the first target data: before loading the sub-target data, estimate the sub-target capacity of the sub-target data; determine whether the sub-target capacity and the current remaining capacity of the target memory satisfy a second preset condition; if it is determined that the sub-target capacity and the current remaining capacity of the target memory satisfy the second preset condition, load the sub-target data into the target memory; if it is determined that the sub-target capacity and the current remaining capacity of the target memory do not satisfy the second preset condition, stop executing the first loading operation, and determine the data already loaded into the target memory as the first target data.

[0075] In an optional embodiment, the apparatus further includes a merging module, configured to merge multiple elastic distributed datasets (RDDs) included in the target memory after performing a second loading operation on the second target data based on the target memory capacity, wherein the second target data includes multiple RDDs.

[0076] In an optional embodiment, the above apparatus further includes: a third loading module, configured to, after performing a second loading operation on the second target data based on the current remaining capacity of the target memory, and if it is determined that the data in the target memory has been updated, load the updated data in the target memory with the same time span as the first target data into the target memory according to a second preset rule.

[0077] In an optional embodiment, the above apparatus further includes: a third execution module, configured to perform the first loading operation on the third target data stored in the target memory according to the first preset rule after performing the second loading operation on the second target data based on the current remaining capacity of the target memory, and upon receiving a target instruction, wherein the target instruction is an instruction triggered after resetting the time span configuration information.

[0078] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0079] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0080] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0081] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0082] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0083] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0084] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0085] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of loading data, characterized by, The application is applied in a big data application scenario, including: In the case of receiving a start instruction, performing a first loading operation on the first target data stored in the target memory according to a first preset rule, wherein the start instruction is used to indicate that the target data is loaded into the target memory, and the target data includes the first target data; In the case of determining that the first loading operation is completed, performing a second loading operation on the second target data based on the current remaining capacity of the target memory, wherein the second target data includes the incremental data of the data currently stored in the target memory relative to the data stored in the target memory before the first loading operation is performed; Performing a first loading operation on the first target data stored in the target memory according to a first preset rule includes: obtaining time span configuration information, wherein the time span configuration information is used to indicate the time span information of generating the target data; based on the time span configuration information, determining the time span granularity of the first loading operation; dividing the first target data into one or more sub-target data according to the time span granularity; sequentially performing the first loading operation on one or more sub-target data in the order from back to front in units of the time span granularity, so as to load the first target data into the target memory; Sequentially performing the first loading operation on one or more sub-target data in the order from back to front in units of the time span granularity includes: in the case that the first target data is not loaded and the target memory reaches a preset threshold, stopping performing the first loading operation, and re-determining the data loaded into the target memory as the first target data, wherein the preset threshold is a predetermined proportion of the capacity of the target memory.

2. The method of claim 1, wherein, Performing a second loading operation on the second target data based on the current remaining capacity of the target memory includes: Determining the target capacity required by the second target data; Judging whether the first preset condition is met between the target capacity and the current remaining capacity of the target memory; In the case that it is determined that the first preset condition is met between the target capacity and the current remaining capacity of the target memory, performing the second loading operation on the second target data.

3. The method of claim 2, wherein, After judging whether the first preset condition is met between the target capacity and the current remaining capacity of the target memory, the method further includes: In the case that it is determined that the first preset condition is not met between the target capacity and the current remaining capacity of the target memory, continuously cleaning the target memory in a preset manner; In the case that it is determined that the first preset condition is met between the target capacity and the remaining capacity of the target memory after cleaning, performing the second loading operation on the second target data.

4. The method of claim 3, wherein, Continuously cleaning the target memory in a preset manner includes at least one of the following: Continuously cleaning the target memory according to a predetermined time span granularity; Continuously cleaning the target memory according to a predetermined time sequence.

5. The method according to any one of claims 1-4, characterized in that, Performing a second loading operation on the second target data based on the current remaining capacity of the target memory includes: performing a second loading operation on second target data according to a preset period based on a current remaining capacity of the target memory.

6. The method of claim 1, further comprising: after performing a first loading operation on first target data stored in a target memory according to a first preset rule, the method further comprises: recording first time span information of performing the first loading operation on the first target data, wherein a deadline included in the first time span information is a time of starting to perform the first loading operation; after performing a second loading operation on second target data based on a current remaining capacity of the target memory, the method further comprises:

7. The method of claim 1, wherein, recording second time span information of performing the second loading operation on the second target data, wherein a starting time included in the second time span information is the deadline included in the first time span information. the method further comprises: for each of the sub-target data included in the first target data, the following operations are performed: before loading the sub-target data, estimating a sub-target capacity of the sub-target data; determining whether the sub-target capacity and the current remaining capacity of the target memory satisfy a second preset condition; in a case where it is determined that the sub-target capacity and the current remaining capacity of the target memory satisfy the second preset condition, loading the sub-target data into the target memory; 8. The method of claim 1, wherein, in a case where it is determined that the sub-target capacity and the current remaining capacity of the target memory do not satisfy the second preset condition, stopping performing the first loading operation, and re-determining data that has been loaded into the target memory as the first target data. after performing a second loading operation on second target data based on a target memory capacity, the method further comprises:

9. The method of claim 1, wherein, merging multiple resilient distributed datasets (RDDs) included in the target memory, the second target data including multiple RDDs. after performing a second loading operation on second target data based on a current remaining capacity of the target memory, the method further comprises:

10. The method of claim 1, wherein, in a case where it is determined that there is an update in the data in the target memory, loading data included in the target memory and having a same time span as the first target data into the target memory according to a second preset rule. after performing a second loading operation on second target data based on a current remaining capacity of the target memory, the method further comprises:

11. A data loading apparatus, characterized by comprising: in a case where a target instruction is received, performing the first loading operation on third target data stored in the target memory according to the first preset rule, wherein the target instruction is an instruction triggered after resetting time span configuration information. application in a big data application scenario, comprising: a first loading module configured to, in a case where a start instruction is received, perform a first loading operation on first target data stored in a target memory according to a first preset rule, wherein the start instruction is used to instruct loading target data into a target memory, and the target data includes the first target data; a second loading module, configured to, when it is determined that the first loading operation is completed, perform a second loading operation on second target data based on the current remaining capacity of the target memory, wherein the second target data comprises incremental data of data currently stored in the target memory relative to data stored in the target memory before the first loading operation is performed; the first loading module comprises: an obtaining unit, configured to obtain time span configuration information, wherein the time span configuration information is used to indicate time span information of the target data; a second determining unit, configured to determine a time span granularity of the first loading operation based on the time span configuration information; a dividing unit, configured to divide the first target data into one or more sub-target data according to the time span granularity; and a third executing unit, configured to perform the first loading operation on the one or more sub-target data in a sequence from back to front in a unit of the time span granularity, so as to load the first target data into the target memory. the device is further configured to, when the first target data is not loaded completely and the target memory reaches a preset threshold, stop performing the first loading operation, and re-determine data loaded into the target memory as the first target data, wherein the preset threshold is a predetermined proportion of the capacity of the target memory.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein the computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 10.

13. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 10.

Images