Data loading method and device, electronic equipment and storage medium

Abstract

The application provides a data loading method and device, electronic equipment and storage medium. The method comprises the following steps: obtaining a data loading request of a processing unit, and determining a data amount of to-be-loaded data based on the data loading request; in the case that the data amount of the to-be-loaded data is greater than a current available storage space size of a memory, determining that a target loading position of the to-be-loaded data is a target disk space; determining a target directory in the target disk space, so that the processing unit performs a lock operation on the target directory, and based on the lock operation, the to-be-loaded data is written into a file in the target directory in the mode of exclusively occupying the target directory at the same time. The data loading method and device, electronic equipment and storage medium provided by the application realize reasonable planning of a data loading position, and meanwhile ensure stable operation of a data loading process.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data loading method, apparatus, electronic device, and storage medium. Background Technology

[0002] Computer storage resources can be broadly divided into two types: RAM and disk. In terms of storage space, the usable space of a disk is much larger than that of RAM; however, in terms of computer access speed, the access speed to RAM is approximately 100,000 times faster than the access speed to a disk; that is, RAM has a fast access speed but a small storage space, while disk has a slow access speed but a large storage space.

[0003] A computer typically has multiple independent processing units operating simultaneously. Most of the data processing and computation by these units is done by writing data into memory, completing the loading process. However, memory resources are limited. As the number of computer processing units and the amount of data processed increases, memory resources become scarce, necessitating the use of disk resources for loading. Currently, there is no complete technical solution for selecting the appropriate loading location for specific data to be processed. Summary of the Invention

[0004] The purpose of this invention is to provide a data loading method, apparatus, electronic device, and storage medium to solve the technical problems existing in the prior art.

[0005] This invention provides a data loading method, comprising:

[0006] Obtain the data loading request from the processing unit, and determine the amount of data to be loaded based on the data loading request;

[0007] If the amount of data to be loaded is greater than the current available storage space in memory, the target loading location of the data to be loaded is determined to be the target disk space; wherein, the current available storage space in memory is obtained by real-time statistics on memory usage;

[0008] A target directory is determined within the target disk space, so that the processing unit performs a locking operation on the target directory, and based on the locking operation, writes the data to be loaded into the file in the target directory at the same time in a manner that exclusively owns the target directory.

[0009] According to the data loading method provided by the present invention, determining the target directory within the target disk space includes:

[0010] Determine a target directory from the candidate directories according to the data volume of the data to be loaded and the upper limit value of the logical storage space of each candidate directory in the target disk space; wherein the target disk space has at least one candidate directory.

[0011] The data loading method provided by the application further comprises:

[0012] In the case that the data volume of the data to be loaded is less than or equal to the current available storage space size of the memory, the target loading position of the data to be loaded is determined as the memory.

[0013] The application further provides a data loading method, comprising:

[0014] According to the data volume of the data to be loaded, a data loading request is generated and sent, so that the operating system determines the target loading position of the data to be loaded as the target disk space in the case that the data volume of the data to be loaded is greater than the current available storage space size of the memory, and determines a target directory in the target disk space; wherein the current available storage space size of the memory is obtained by real-time statistics on the usage of the memory;

[0015] Receive the information of the target directory, perform a lock operation on the target directory, and based on the lock operation, write the data to be loaded into a file in the target directory in the manner of exclusively occupying the target directory at the same time.

[0016] According to the data loading method provided by the application, the lock operation on the target directory and the writing of the data to be loaded into a file in the target directory based on the lock operation at the same time in the manner of exclusively occupying the target directory comprise:

[0017] Perform a first lock operation on the target directory, and based on the first lock operation, obtain the unique control right of the target directory;

[0018] Create a target data file in the target directory, and write the data to be loaded into the target data file;

[0019] Perform a first unlock operation on the target directory, and perform a second lock operation on the target data file, and based on the second lock operation, obtain the unique control right of the target data file;

[0020] After the processing of the target data file is completed, perform a second unlock operation on the target data file, and delete the target data file.

[0021] According to the data loading method provided by the application, after the first lock operation on the target directory, the method further comprises:

[0022] performing a second lock operation on the existing data file in the target directory;

[0023] deleting the existing data file on which the second lock operation can be performed.

[0024] The application further provides a data loading device, comprising:

[0025] an obtaining module, configured to obtain a data loading request of a processing unit and determine a data amount of to-be-loaded data based on the data loading request;

[0026] a selecting module, configured to determine a target loading position of the to-be-loaded data as a target disk space in a case where the data amount of the to-be-loaded data is greater than a current available storage space size of a memory; wherein the current available storage space size of the memory is obtained by performing real-time statistics on a usage of the memory;

[0027] a loading module, configured to determine a target directory in the target disk space, so that the processing unit performs a lock operation on the target directory, and based on the lock operation, write the to-be-loaded data into a file in the target directory in a manner of exclusively occupying the target directory at the same time.

[0028] The application further provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the communication handshake method based on the TLS protocol according to any one of the above when executing the program.

[0029] The application further provides a non-transitory computer readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the steps of the data loading method according to any one of the above.

[0030] The application further provides a computer program product, which comprises a computer program, wherein the computer program is executable on a processor to implement the steps of the data loading method according to any one of the above.

[0031] The data loading method, device, electronic equipment and storage medium provided by the application, by taking the data amount of the data to be loaded as the basis for selecting the data loading position, specifically selecting whether the memory or the disk is the data loading position, and based on this, the first step of planning for the data loading position selection is realized; when the disk is selected as the data loading position, the uniqueness of the data loading process under the single disk directory is realized by performing the lock operation on the disk directory, and based on this, the second step of planning for the data loading position selection is realized; based on the above planning process, the reasonable planning of the data loading process is realized, which is beneficial to ensure the stable operation of the data loading process and avoid the problems of interruption of the data loading process or disk crash. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0033] Figure 1 is one of the flowcharts of the data loading method provided by the application;

[0034] Figure 2 is the second flowchart of the data loading method provided by the application;

[0035] Figure 3 is the computer storage resource structure relationship diagram provided by the application;

[0036] Figure 4 is the structure diagram of the data loading method provided by the application;

[0037] Figure 5 is the structure diagram of the data acquisition from the network provided by the application;

[0038] Figure 6 is the structure diagram of the data acquisition from the disk file directory provided by the application;

[0039] Figure 7 is the structure diagram of the simultaneous access of the multiple processing units to the single disk directory provided by the application;

[0040] Figure 8 is the structure diagram of the mutual exclusion access of the multiple processing units to the single disk directory provided by the application;

[0041] Figure 9 is the structure diagram of the parallel access of the multiple processing units to the multiple disk directories provided by the application;

[0042] Figure 10is a structural schematic diagram of a data loading device provided by the present application.

[0043] Figure 11 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0044] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0045] Figure 1 is one of flowcharts of a data loading method provided by the present application, which is applied to an operating system, such as Figure 1 As shown in the figure, the method comprises the following steps.

[0046] S110, obtaining a data loading request of a processing unit, and determining a data amount of to-be-loaded data based on the data loading request;

[0047] S120, in a case where the data amount of the to-be-loaded data is greater than a current available storage space size of the memory, determining a target loading position of the to-be-loaded data as a target disk space; wherein the current available storage space size of the memory is obtained by real-time statistics on a usage of the memory;

[0048] S130, determining a target directory in the target disk space, so that the processing unit performs a lock operation on the target directory, and based on the lock operation, writes the to-be-loaded data into a file in the target directory in a manner of exclusively occupying the target directory at the same time.

[0049] Figure 3 is a structural relationship diagram of computer storage resources provided by the present application, as shown in the figure. Figure 3 The computer storage resources comprise a memory and a disk, and there are multiple independent processing units, also called processing processes, on one computer. Different processing units process and calculate data, and most of the data is loaded into the memory for processing. However, the memory resource is limited, and with the increase of the computer processing unit and the processing data amount, the problem of insufficient memory resource will occur, at which time the disk resource needs to be used for storage. The memory has the characteristics of fast data loading speed, small space and non-persistent data, while the disk has the characteristics of slow data loading speed, large space and persistent data.

[0050] It should be noted that the data amount size of the to-be-loaded data is first acquired through the data loading request, and the data amount size of the to-be-loaded data is taken as the selection basis of the data loading position, specifically: when the data amount size of the to-be-loaded data is greater than the current available storage space of the memory, the loading position of the to-be-loaded data is selected as the disk; for example, the memory size of a computer is 4G, because other multiple processing units have been loaded on the memory at this time, so the available storage space size of the memory is 500M at this time, but the data amount size of the to-be-loaded data is 600M, because the data amount size of the to-be-loaded data is already greater than the available storage space of the current memory, so the loading position of the to-be-loaded data is selected as the disk at this time.

[0051] After the loading position of the to-be-loaded data is selected as the disk, the specific data loading process is executed, specifically: a target directory is determined in the target disk space, then the to-be-loaded data is written into the target directory, and a lock operation is performed on the target directory in the data writing process, here the lock operation refers to locking the target directory by using a file lock, to ensure the uniqueness of the to-be-loaded data in the target directory data loading process. It should be noted that the data loading process of the to-be-loaded data in the target directory includes the data writing process and the data processing process after the data writing is completed, and the locking state of the target directory runs through the entire data loading process, that is, the target directory always remains in the locked state during the data writing process and the data processing process.

[0052] The data loading method provided by the application selects the data loading position as the memory or the disk based on the data amount size of the to-be-loaded data, realizes the first step planning of the data loading position selection, realizes the uniqueness of the data loading process in a single disk directory by performing the lock operation on the disk directory when the disk is selected as the data loading position, realizes the second step planning of the data loading position selection, realizes the reasonable planning of the data loading process based on the above planning process, and is beneficial to ensure the stable operation of the data loading process and avoid the problems of data loading process interruption or disk crash.

[0053] According to the data loading method provided by the application, in the application, the target directory is determined in the target disk space, including: determining the target directory from the each candidate directory according to the data amount of the to-be-loaded data and the logical storage space upper limit value of each candidate directory in the target disk space; wherein the target disk space has at least one candidate directory.

[0054] It should be noted that the disk is pre-divided into multiple subdirectories, and each subdirectory is pre-provided with a logical space upper limit value (a logical value artificially specified), that is, the maximum amount of data allowed to be stored in each subdirectory. The sum of the space upper limit values of all subdirectories is less than or equal to the actual size of the disk space. Based on the setting process, as long as no data exceeding the logical space upper limit value is stored under each subdirectory, it can be ensured that the entire disk will not have a data overflow problem. When creating subdirectories for the disk, the space upper limit value of each subdirectory and the number of subdirectories need to be pre-set according to actual needs. Taking the existing business scenario as an example: in the current business scenario, the super-large data to be processed is generally within 5G, and the size of the extremely rare data is between 5G and 10G, and it is extremely rare to have data reaching the 10G-20G interval. Assuming that the existing space size of a specific disk is 100G, under such limited conditions, the disk can be divided into 15 subdirectories, numbered 1 to 15, wherein the logical space of subdirectories 1 to 12 is set to 5G, the logical space of subdirectories 13 to 14 is set to 10G, and the logical space of subdirectory 15 is set to 20G. Thus, the 100G space is just used up. The reason for performing the above subdirectory creation process on the 100G disk is that most of the data to be processed is within 5G, a small part is within 5G-10G, and a very small part is within 10G-20G.

[0055] Based on the above-described subdirectory creation process for the 100G disk, when the data amount of the data to be loaded is 2G, theoretically, the above-mentioned 15 subdirectories can all implement the loading process of the data to be loaded, and when the data amount of the data to be loaded is 6G, only one of the 13th-15th subdirectories can be selected to implement the data loading process.

[0056] The data loading method provided by the application takes the data amount of the data to be loaded as the selection premise and takes the logical storage space upper limit value of each subdirectory in the disk as the selected basis, thereby realizing the optimal selection of the data to be loaded in a certain subdirectory of the disk, ensuring the stable operation of the disk and avoiding disk crashes, and realizing the parallel processing of multiple processing units, thereby effectively improving the reliability and efficiency of data loading.

[0057] According to the data loading method provided by the application, in the application, the method further comprises: in the case that the data amount of the data to be loaded is less than or equal to the current available storage space size of the memory, determining that the target loading position of the data to be loaded is the memory.

[0058] It should be noted that compared with the disk, the data loading in the memory has the advantage of fast loading speed, so when the data amount of the to-be-loaded data is less than or equal to the current available storage space of the memory, the memory is preferentially selected as the loading position of the to-be-loaded data, and an efficient loading process of the to-be-loaded data is realized.

[0059] The data loading method provided by the application compares the data amount of the to-be-loaded data with the current available storage space of the memory, and preferentially selects the memory as the loading position of the to-be-loaded data when the data amount of the to-be-loaded data is less than or equal to the current available storage space of the memory, thereby realizing an efficient loading process of the to-be-loaded data.

[0060] Figure 2 Fig. 2 is a flowchart of a data loading method provided by the application, which is applied to a processing unit, such as Figure 2 As shown in the figure, the method comprises the following steps:

[0061] S210, generating and sending a data loading request according to the data amount of the to-be-loaded data, so that the operating system determines that the target loading position of the to-be-loaded data is a target disk space and determines a target directory in the target disk space when the data amount of the to-be-loaded data is greater than the current available storage space of the memory; wherein the current available storage space of the memory is obtained by real-time statistics on the usage of the memory.

[0062] S220, receiving the information of the target directory, performing a lock operation on the target directory, and based on the lock operation, writing the to-be-loaded data into a file in the target directory in a manner of exclusively occupying the target directory at the same time.

[0063] It should be noted that the current available storage space of the memory represents the real-time occupation of the memory, specifically: the specification value of the memory-the real-time occupation value of the memory = the current available storage space of the memory, and the current available storage space of the memory can be used to obtain the current usage of the memory in real time. After performing a lock operation on a target directory in the disk, the to-be-loaded data loaded under the target directory is unique, and only when the data loading is completed, an unlock operation is performed, and then the data loading process of the next to-be-loaded data is entered.

[0064] The data loading method provided by the application selects the data loading position as the memory or the disk according to the data amount of the data to be loaded, and thus the first step of the data loading position selection is planned; when the disk is selected as the data loading position, the lock operation is performed on the disk directory to ensure the uniqueness of the data loading process in the single disk directory, and thus the second step of the data loading position selection is planned; based on the above planning process, the data loading process is reasonably planned, which is beneficial to ensure the stable operation of the data loading process and avoid the interruption of the data loading process or the disk crash.

[0065] According to the data loading method provided by the application, the target directory is locked, and the data to be loaded is written into the file in the target directory in the exclusive mode of the target directory at the same time based on the lock operation.

[0066] The target directory is locked, and the unique control right of the target directory is obtained based on the first lock operation; the target data file is created in the target directory, and the data to be loaded is written into the target data file; the first unlock operation is performed on the target directory, and the second lock operation is performed on the target data file, and the unique control right of the target data file is obtained based on the second lock operation; after the processing of the target data file is completed, the second unlock operation is performed on the target data file, and the target data file is deleted.

[0067] It should be noted that in the embodiment of the application, there are two file locks in the data loading process, one of which is the target directory file lock, which is used to perform the first lock operation, that is, when the data to be loaded is written into the target directory in the disk, the first lock operation is performed on the target directory, and the unique control right of the target directory is obtained based on the operation, that is, the data to be loaded written in the target directory is unique, which ensures that multiple data to be loaded is not written at the same time, and the unique control right of the target directory is obtained based on the first lock operation, and the target data file is created in the target directory, and the data to be loaded is written into the target data file.

[0068] The other file lock is the target data file lock, which is used to perform the second lock operation, and when the data to be loaded is written into the target data file, the first unlock operation is performed on the target directory file lock, and the second lock operation is performed on the target data file, and the unique control right of the target data file is obtained based on the second lock operation, and the unique control right of the target data file is reflected in that the processing unit corresponding to the target data file is fixed, which ensures the uniqueness of the processing process of the data to be loaded in the target data file.

[0069] After the processing of the target data file is completed, i.e. the loading of the to-be-loaded data in the target data file is completed, a second unlocking operation is performed on the target data file, and the target data file is deleted.

[0070] The data loading method provided by the application realizes the uniqueness of the to-be-loaded data for the target directory using process through the first locking operation of the target directory file lock, realizes the planning control of the to-be-loaded data writing process, and avoids the problem of disk collapse; meanwhile, the to-be-loaded data realizes the uniqueness of the data processing process through the second locking operation of the target data file lock, i.e. the processing unit for the to-be-loaded data is fixed, and the reliability and stability of the to-be-loaded data processing process are ensured; based on the above process, the stable and reliable operation of the entire process from the data writing to the data processing of the to-be-loaded data is realized, the reliable loading process of the to-be-loaded data is realized while the stable operation of the disk is ensured.

[0071] According to the data loading method provided by the application, after the first locking operation on the target directory is performed, a second locking operation is further performed on the existing data file in the target directory; in the case that the existing data file can be subjected to the second locking operation, the existing data file that can be subjected to the second locking operation is deleted.

[0072] It should be noted that after the to-be-loaded data is processed by the processing unit, the data needs to be deleted, and the situation of false deletion or omission should not occur. Normally, after the to-be-loaded data is processed, the to-be-loaded data is immediately deleted, and the use right of the target directory is released; meanwhile, abnormal situations may also exist, specifically: when a processing unit holds the control right of a target directory, and the to-be-loaded data has been successfully written into or is being written into the target directory, the processing unit directly exits due to program abnormality, resulting in residual data in the target directory, and the disk cannot automatically clean up the residual data. The abnormal situation can be divided into two kinds: the to-be-loaded data has been successfully written into the target directory, but the program abnormally exits during the processing of the data after the writing is completed, i.e. an abnormality occurs in the processing of the to-be-loaded data; the to-be-loaded data abnormally exits during the writing into the target directory, i.e. an abnormality occurs in the writing process of the to-be-loaded data.

[0073] Based on the problem that residual data caused by the above two abnormal conditions cannot be automatically cleared, in the embodiment of the application, after the first lock operation is performed on the target directory, a second lock operation is performed on the existing data file in the target directory, if the second lock succeeds, it indicates that there is residual data in the target directory, and then a delete operation is performed on the residual data; if the second lock fails, it indicates that there is no residual data in the target directory, and then a delete operation is not needed to be performed, and a target data file is created in the target directory in time to realize subsequent writing and processing of the to-be-loaded data.

[0074] The data loading method provided by the application realizes the determination of whether there is residual data in the target directory by using the target data file lock to perform the lock operation on the existing data file in the target directory, and realizes the immediate deletion of the residual data on the premise that there is residual data in the target directory, based on which, the real-time cleaning processing of the disk is realized, and then the maximum use of the disk space is ensured.

[0075] Figure 4 is a structural schematic diagram of the data loading method provided by the application, as shown in Figure 4 The upper limit value of the available memory space of each processing unit is 500M, the existing processing units 1, 2, 3, 4 and 5, in which the processing units 4 and 5 need to process 100M and 200M of data respectively, the processing process is directly loaded into the memory for fast processing, but the processing units 1, 2 and 3 need to process 1G, 2G and 3G of super large data respectively, at this time, if the data is directly loaded into the memory for data processing, it will cause the problem of memory crash, and then the processing unit is directly recycled by the operating system, based on the above reason, the data loading process of the processing unit 1, the processing unit 2 and the processing unit 3 is loaded into the disk for processing, based on which, the problem of memory crash is avoided.

[0076] For a processing unit, the data to be processed mainly has two sources, one is to obtain data by network downloading, and the other is to obtain data by accessing other disk file directories; Figure 5 is a structural schematic diagram of obtaining data from the network provided by the application, Figure 6 is a structural schematic diagram of obtaining data from the disk file directory provided by the application.

[0077] Figure 7 is a structural schematic diagram of multiple processing units simultaneously accessing a single disk directory provided by the application, as shown in Figure 7As shown, the existing 1, 2, 3 processing units, and the data size they will write to the disk is 3G, 2G, 1G in turn. 1, 2, 3 represent the operation sequence of each processing unit to write data. Assuming that the size of a specific disk directory is: 5G space, 1 data arrives first, wants to write 3G data, starts to write data to the disk and has successfully written 500M data, at this time the statistics show that the remaining available space is 4.5G, but at this time 2, 3 arrive in turn, want to write 2G, 1G data to the disk, and the real-time monitoring statistics show that the remaining disk space is 4.5G, so they all start to write data to the disk, but the real-time monitoring statistics are misleading, because when processing unit 1 is completely written, it will actually occupy 3G of space, and the remaining space is only 2G, at this time processing unit 2 and processing unit 3 mistakenly believe that the disk has enough storage space, and finally when processing unit 1, processing unit 2 and processing unit 3 completely write data to the disk, the disk will crash.

[0078] Based on the above problems, in the embodiment of the present application, the following processing is performed, specifically: let each processing unit mutually exclusive access to this directory, that is, processing unit 1 accesses, 2 and 3 cannot access and can only wait. Figure 8 It is a structure diagram of the mutual access of multiple processing units to a single disk directory provided by the present application, as shown in Figure 8 As shown, a file is created in the disk directory using file locking, such as creating a filelock file, which is used to realize the function of file locking, and the file lock is used to lock the control right of the processing unit in the directory. When processing unit 1, processing unit 2 and processing unit 3 try to lock the file filelock, if they can successfully lock, it is considered that the processing unit has grabbed the operation right of the disk directory where the file is located, and other processing units cannot operate on the disk directory.

[0079] Figure 9 It is a structure diagram of the parallel access of multiple processing units to multiple disk directories provided by the present application, as shown in Figure 9 As shown, multiple subdirectories are created in the disk directory, and a filelock is created in each subdirectory to lock the subdirectory, so that each subdirectory can be accessed by only one processing unit at the same time. Based on this setting, multiple subdirectories exist in a disk at the same time, and multiple processing units can simultaneously grab the file lock and access, that is, the parallel processing of multiple processing units is realized. Similarly, the processing units that do not grab the lock need to wait for the release of the lock.

[0080] Figure 10 It is a structure diagram of a data loading device 1000 provided by the present application, as shown in Figure 10 As shown, the device comprises:

[0081] The acquisition module 1010 is configured to acquire a data loading request of a processing unit, and determine a data amount of to-be-loaded data based on the data loading request.

[0082] The selection module 1020 is configured to determine that a target loading position of the to-be-loaded data is a target disk space when the data amount of the to-be-loaded data is greater than a current available storage space size of the memory.

[0083] The loading module 1030 is configured to determine a target directory in the target disk space, so that the processing unit performs a lock operation on the target directory, and based on the lock operation, writes the to-be-loaded data into a file in the target directory in a manner of exclusively occupying the target directory at the same time.

[0084] The data loading device provided by the application selects the data loading position as the memory or the disk according to the data amount of the to-be-loaded data, and thus the first step of the data loading position selection is realized. When the disk is selected as the data loading position, the uniqueness of the data loading process in a single disk directory is realized by performing the lock operation on the disk directory, and thus the second step of the data loading position selection is realized. Based on the above planning process, the data loading process is reasonably planned, which is beneficial to ensure the stable operation of the data loading process and avoid the interruption of the data loading process or the disk crash.

[0085] According to the data loading device provided by the application, when the loading module 1030 is used to determine the target directory in the target disk space, the loading module 1030 is specifically used to:

[0086] The target directory is determined from the candidate directories according to the data amount of the to-be-loaded data and the upper limit value of the logical storage space of each candidate directory in the target disk space.

[0087] The data loading device provided by the application selects the data loading position as the memory or the disk according to the data amount of the to-be-loaded data, and thus the first step of the data loading position selection is realized. When the disk is selected as the data loading position, the uniqueness of the data loading process in a single disk directory is realized by performing the lock operation on the disk directory, and thus the second step of the data loading position selection is realized. Based on the above planning process, the data loading process is reasonably planned, which is beneficial to ensure the stable operation of the data loading process and avoid the interruption of the data loading process or the disk crash.

[0088] According to the data loading device provided in the application, in the application, the device further comprises a determination module configured to determine that the target loading position of the to-be-loaded data is the memory when the data volume of the to-be-loaded data is less than or equal to the current available storage space size of the memory.

[0089] The data loading device provided in the application compares the data volume of the to-be-loaded data with the current available storage space of the memory, and preferentially selects the memory as the loading position of the to-be-loaded data when the data volume of the to-be-loaded data is less than or equal to the current available storage space of the memory, thereby realizing an efficient loading process of the to-be-loaded data.

[0090] Figure 11 An example of a schematic diagram of a physical structure of an electronic device is shown in Figure 11 The electronic device can include a processor 1110, a communications interface 1120, a memory 1130, and a communications bus 1140, wherein the processor 1110, the communications interface 1120, and the memory 1130 can communicate with each other through the communications bus 1140. The processor 1110 can invoke a logical instruction in the memory 1130 to execute a data loading method, which includes obtaining a data loading request of a processing unit and determining a data volume of to-be-loaded data based on the data loading request; determining that a target loading position of the to-be-loaded data is a target disk space when the data volume of the to-be-loaded data is greater than a current available storage space size of a memory; wherein the current available storage space size of the memory is obtained by performing real-time statistics on a usage of the memory; determining a target directory in the target disk space, so that the processing unit performs a lock operation on the target directory, and based on the lock operation, writes the to-be-loaded data into a file in the target directory in a manner of exclusively occupying the target directory at the same time.

[0091] or executing a data loading method, which includes generating and sending a data loading request according to a data volume of to-be-loaded data, so that an operating system determines that a target loading position of the to-be-loaded data is a target disk space when the data volume of the to-be-loaded data is greater than a current available storage space size of a memory, and determines a target directory in the target disk space; wherein the current available storage space size of the memory is obtained by performing real-time statistics on a usage of the memory; receiving information of the target directory, performing a lock operation on the target directory, and based on the lock operation, writing the to-be-loaded data into a file in the target directory in a manner of exclusively occupying the target directory at the same time.

[0092] Further, the logic instructions in the memory 1130 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0093] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions, when the program instructions are executed by a computer, the computer can execute the data loading method provided by the above-mentioned methods, and the method comprises: obtaining a data loading request of a processing unit, and determining a data amount of to-be-loaded data based on the data loading request; in a case where the data amount of the to-be-loaded data is greater than a current available storage space size of a memory, determining a target loading position of the to-be-loaded data as a target disk space; wherein the current available storage space size of the memory is obtained by real-time statistics on a usage of the memory; determining a target directory in the target disk space, so that the processing unit performs a lock operation on the target directory, and based on the lock operation, writes the to-be-loaded data into a file in the target directory in a manner of exclusively occupying the target directory at the same time.

[0094] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions, when the program instructions are executed by a computer, the computer can execute the data loading method provided by the above-mentioned methods, and the method comprises: obtaining a data loading request of a processing unit, and determining a data amount of to-be-loaded data based on the data loading request; in a case where the data amount of the to-be-loaded data is greater than a current available storage space size of a memory, determining a target loading position of the to-be-loaded data as a target disk space; wherein the current available storage space size of the memory is obtained by real-time statistics on a usage of the memory; determining a target directory in the target disk space, so that the processing unit performs a lock operation on the target directory, and based on the lock operation, writes the to-be-loaded data into a file in the target directory in a manner of exclusively occupying the target directory at the same time.

[0095] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the data loading method provided above, which comprises: obtaining a data loading request of a processing unit, and determining a data amount of data to be loaded based on the data loading request; in a case where the data amount of the data to be loaded is greater than a current available storage space size of a memory, determining a target loading position of the data to be loaded as a target disk space; wherein the current available storage space size of the memory is obtained by real-time statistics on a usage of the memory; determining a target directory in the target disk space, so that the processing unit performs a lock operation on the target directory, and based on the lock operation, writes the data to be loaded into a file in the target directory in a manner of exclusively occupying the target directory at the same time.

[0096] or executes the data loading method, which comprises: generating and sending a data loading request according to a data amount of data to be loaded, so that an operating system determines a target loading position of the data to be loaded as a target disk space in a case where the data amount of the data to be loaded is greater than a current available storage space size of a memory, and determines a target directory in the target disk space; wherein the current available storage space size of the memory is obtained by real-time statistics on a usage of the memory; receiving information of the target directory, performing a lock operation on the target directory, and based on the lock operation, writing the data to be loaded into a file in the target directory in a manner of exclusively occupying the target directory at the same time.

[0097] The device embodiments described above are merely illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0098] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0099] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features therein can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data loading method, characterized by, The method comprises: acquiring a data loading request of a processing unit and determining a data amount of data to be loaded based on the data loading request; in a case where the data amount of the data to be loaded is greater than a current available storage space size of a memory, determining a target loading position of the data to be loaded as a target disk space; wherein the current available storage space size of the memory is obtained by real-time statistics on a usage of the memory; determining a target directory in the target disk space, so that the processing unit performs a lock operation on the target directory, and based on the lock operation, writes the data to be loaded into a file in the target directory in a manner of exclusively occupying the target directory at the same time; the lock operation on the target directory and the writing of the data to be loaded into the file in the target directory in the manner of exclusively occupying the target directory at the same time based on the lock operation comprise: performing a first lock operation on the target directory, and based on the first lock operation, obtaining unique control over the target directory; creating a target data file in the target directory and writing the data to be loaded into the target data file; performing a first unlock operation on the target directory and a second lock operation on the target data file, and based on the second lock operation, obtaining unique control over the target data file; after completing the processing of the target data file, performing a second unlock operation on the target data file and deleting the target data file; the determination of the target directory in the target disk space comprises: determining the target directory from each candidate directory in the target disk space according to the data amount of the data to be loaded and an upper limit value of a logical storage space of each candidate directory; wherein the target disk space has at least one candidate directory; after the first lock operation on the target directory, the method further comprises: performing a second lock operation on an existing data file in the target directory; in a case where the existing data file can be subjected to the second lock operation, deleting the existing data file that can be subjected to the second lock operation.

2. The data loading method of claim 1, wherein, The method further comprises: in a case where the data amount of the data to be loaded is less than or equal to the current available storage space size of the memory, determining a target loading position of the data to be loaded as the memory.

3. A data loading method, characterized by, The method comprises: generating a data loading request according to a data amount of data to be loaded and sending the data loading request, so that an operating system determines a target loading position of the data to be loaded as a target disk space in a case where the data amount of the data to be loaded is greater than a current available storage space size of a memory, and determines a target directory in the target disk space; wherein the current available storage space size of the memory is obtained by real-time statistics on a usage of the memory; receiving information of the target directory, performing a lock operation on the target directory, and based on the lock operation, writing the data to be loaded into a file in the target directory in a manner of exclusively occupying the target directory at the same time; The lock operation is performed on the target directory, and based on the lock operation, the to-be-loaded data is written into a file in the target directory in an exclusive manner at the same time. A first lock operation is performed on the target directory, and based on the first lock operation, unique control of the target directory is obtained. A target data file is created in the target directory, and the to-be-loaded data is written into the target data file. A first unlock operation is performed on the target directory, and a second lock operation is performed on the target data file, and based on the second lock operation, unique control of the target data file is obtained. After the processing of the target data file is completed, a second unlock operation is performed on the target data file, and the target data file is deleted. The target directory is determined in the target disk space, including: determining the target directory from each candidate directory in the target disk space according to the data amount of the to-be-loaded data and the upper limit value of the logical storage space of each candidate directory in the target disk space; wherein the target disk space has at least one candidate directory. After the first lock operation is performed on the target directory, the following operations are further included: A second lock operation is performed on an existing data file in the target directory. In the case that the existing data file can be executed by the second lock operation, the existing data file capable of being executed by the second lock operation is deleted.

4. A data loading apparatus, characterized by comprising: Including: An acquisition module is configured to acquire a data loading request of a processing unit, and determine a data amount of to-be-loaded data based on the data loading request; A selection module is configured to determine a target loading position of the to-be-loaded data as a target disk space in the case that the data amount of the to-be-loaded data is greater than a current available storage space size of a memory; wherein the current available storage space size of the memory is obtained by real-time statistics on the usage of the memory. A loading module is configured to determine a target directory in the target disk space, so that the processing unit performs a lock operation on the target directory, and based on the lock operation, the to-be-loaded data is written into a file in the target directory in an exclusive manner at the same time. The loading module is configured to: Perform a first lock operation on the target directory, and based on the first lock operation, unique control of the target directory is obtained. Create a target data file in the target directory, and write the to-be-loaded data into the target data file. Perform a first unlock operation on the target directory, and perform a second lock operation on the target data file, and based on the second lock operation, unique control of the target data file is obtained. After the processing of the target data file is completed, perform a second unlock operation on the target data file, and delete the target data file. The loading module is further configured to determine the target directory from each candidate directory in the target disk space according to the data amount of the to-be-loaded data and the upper limit value of the logical storage space of each candidate directory in the target disk space; wherein the target disk space has at least one candidate directory. After the first lock operation is performed on the target directory, the method further comprises: performing a second lock operation on an existing data file in the target directory; deleting the existing data file that can be executed by the second lock operation if the existing data file can be executed by the second lock operation.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the data loading method according to any one of claims 1 to 3 when executing the program.

6. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the steps of the data loading method according to any one of claims 1 to 3 when executed by the processor.

7. A computer program product comprising a computer program, characterised in that, The computer program implements the steps of the data loading method according to any one of claims 1 to 3 when executed by the processor.

Images