Multi-version unstructured data storage method for machine learning applications

Abstract

The application discloses a kind of multi-version unstructured data storage methods for machine learning application, specifically include: (1) the distributed storage framework of unstructured multi-source data: before formal storage, multi-source data is handled and judged using multi-version data storage strategy, the optimal storage strategy is selected based on the trade-off of storage space and retrieval time, and the data compression technology based on neural network can be compressed according to different types of data, to improve the storage efficiency of distributed database and the convenience of machine learning application in data access, processing process.(2) multi-version data storage strategy: face the challenge of continuous generation and storage version change of multi-source data.This method can be deployed in distributed unstructured graph database, the storage process is independent, complete and has global perspective, can realize the management and optimization of data storage, suitable for multi-type data source and high-frequency data update scene.

Description

TECHNICAL FIELD

[0001] The application relates to a non-structured data storage method, in particular to a multi-type and multi-version data storage method for machine learning application. BACKGROUND

[0002] In the field of machine learning, data sets are the core of model training and evaluation. With the advancement of technology and the expansion of applications, the types of data sets have become increasingly diverse. For example, in the field of public security intelligent prevention and control, multi-type data such as text, pictures and videos from sensors, cameras, satellites and unmanned aerial vehicles support the supervision analysis and decision-making management of relevant departments. These data not only have a large volume, but also have different structures, and have become an important resource for machine learning applications. However, different types of data require different storage formats and processing methods, and traditional data compression schemes cannot meet the real-time and convenience requirements of machine learning data processing. At the same time, the continuous generation of data also generates multiple versions of data sets that need to be stored at every moment. Directly storing each version not only greatly increases the storage cost, but also brings great challenges to retrieval and implementation. Data version management not only needs to support incremental storage, but also needs to support retrieval and comparison between multiple versions. Therefore, how to efficiently store and access these multi-version non-structured data is one of the technical challenges currently faced by data storage.

[0003] Currently, data storage solutions for machine learning applications mainly focus on the optimization of storage architecture. Distributed file systems (such as HDFS) and object storage systems (such as Amazon S3) are widely used in large-scale data storage to achieve efficient data management and access. However, these storage solutions still face many challenges when dealing with non-structured data. Traditional centralized relational databases cannot meet the storage needs of large-scale non-structured data, especially in high-concurrency and high-throughput scenarios. In addition, comprehensive storage solutions for multi-type data are not yet fully mature and are not convenient for widespread deployment in practical environments. Therefore, a distributed data storage framework is needed that can meet the storage and frequent version update requirements of multi-type data.

[0004] In the face of complex multi-source data storage scenarios for machine learning applications, multi-type and multi-version massive data bring challenges such as rapid increase in storage cost, increased complexity of data retrieval and version management. The present method proposes a multi-version non-structured data storage solution for machine learning application, which adopts the optimal storage strategy for multi-version metadata through a multi-version data storage strategy, implements compression of different types of metadata based on a neural network algorithm, and stores the compressed data in a distributed non-structured graph database, providing an efficient and low-cost non-structured data storage solution for machine learning applications. SUMMARY

[0005] The application is directed to the multi-type and multi-version data storage problem in the machine learning application scenario, and provides an unstructured distributed data storage method in a distributed database, including a distributed storage framework of unstructured multi-source data and a multi-version data storage strategy.

[0006] The multi-version unstructured data storage method for machine learning application includes:

[0007] (1) The distributed storage framework of unstructured multi-source data: The data storage framework is directed to multi-source data to be stored in the database, and realizes the complete data storage process of version control, classification compression and distributed storage according to the characteristics of complex types and frequent version updates of the multi-source data, and specifically includes the implementation of the multi-version data storage strategy, the multi-type data compression algorithm and the distributed graph database storage scheme.

[0008] (2) Multi-version data storage strategy: Since the multi-source data from sensors, cameras and unmanned aerial vehicles is huge and updated frequently, the multi-version data storage strategy comprehensively considers the storage space cost, retrieval time cost, CPU resource consumption cost and network resource consumption cost, and combines the frequency of version update and access to select the optimal storage strategy, i.e., direct storage or incremental storage, for different data versions.

[0009] Further, the distributed storage framework of unstructured multi-source data in step (1) specifically includes:

[0010] (11) A distributed unstructured graph database is created, and a corresponding graph data model is designed, including the nodes of the graph, the relationship of the data, the attributes and the corresponding representations thereof;

[0011] (12) For the multi-source data to be stored in the database, the data is processed and judged through the multi-version data storage strategy, and the optimal storage strategy, i.e., direct storage or incremental storage, is selected based on the costs of storage space and retrieval time and the frequency of data update and access;

[0012] (13) After the strategic decision is made, the data will be classified according to its characteristics such as text, picture, video, etc., and then the corresponding neural network-based compression algorithm will be used to achieve data compression;

[0013] (14) The compressed data will be stored in a distributed unstructured graph database.

[0014] Furthermore, the multi-version data storage strategy in step (2) specifically includes:

[0015] (21) For multi-source data with storage requirements from sensors, cameras, and drones, update and record the version information of these data;

[0016] (22) Calculate the storage space cost of the version data using direct storage and incremental storage. The storage space cost is the storage space overhead required to store the version data. The storage space cost of the direct storage strategy is the storage space size of the version data. The storage space cost of the incremental storage strategy requires calculating the storage space size of all differences in the difference set between the new and old versions. The difference set can be calculated by performing a difference set calculation on the intersection of the new version and the old and new versions.

[0017] (23) Calculate the retrieval time cost of the version data using direct storage and incremental storage. The retrieval time cost is the time cost of obtaining the complete data of the required version from the storage system. The retrieval time cost of the direct storage strategy is the time consumed to find the version data, which is the time consumed by directly searching the node in the graph database. The retrieval of the incremental storage strategy requires first searching the node of the previous version data, and then obtaining the complete data of the new version through the incremental attributes of the previous version data node. Therefore, the retrieval time cost of the incremental storage is the sum of the time consumed by these two processes.

[0018] (24) Calculate the CPU resource consumption cost of direct storage and incremental storage for this version of data. The CPU resource consumption cost is the CPU resource consumed by storing this version of data. The CPU resource consumption cost of the direct storage strategy is the product of the time required to complete operations such as reading, writing, and copying during data storage and the average CPU usage rate. The CPU resource consumption cost of incremental storage is the product of the time required to complete operations such as detecting, calculating, and merging incremental difference data and the average CPU usage rate.

[0019] (25) Calculate the network resource consumption cost of direct storage and incremental storage for this version of data. The network resource consumption cost is the network bandwidth consumed by storing this version of data. The network resource consumption cost of the direct storage strategy is the quotient of the amount of complete data transmitted during the storage process and the available network bandwidth. The network resource consumption cost of the incremental storage strategy is the quotient of the amount of differential data transmitted during the storage process and the available network bandwidth.

[0020] (26) Calculate the total cost of the version data using direct storage and incremental storage, and the total cost is represented by C. The total cost of the two storage strategies is calculated as the weighted sum of the storage space cost, retrieval time cost, CPU resource consumption cost, and network resource consumption cost. The sum of the weight factors of the four costs is 1. In the data version storage decision, the storage cost and retrieval cost are given priority, and the proportion of each cost in the total cost can be adjusted according to the actual situation;

[0021] (27) Calculate the version update frequency, that is, the frequency of each version data being updated. The calculation method is the quotient of the number of times the data is updated in a certain time interval and the time difference. In addition, a version update frequency threshold is set according to the actual situation.

[0022] (28) Calculate the version access frequency, that is, the frequency of each version data being accessed. The calculation method is the quotient of the number of times the data is accessed in a certain time interval and the time difference. In addition, a version access frequency threshold is set according to the actual situation.

[0023] (29) Consider factors such as the total cost of direct storage and incremental storage, version update frequency, and version access frequency to make a decision on the data version storage strategy and select the optimal storage method. Specifically, if the total cost of direct storage is greater than the total cost of incremental storage, the data update frequency is less than its threshold, and the data access frequency is also less than its threshold, then select the incremental storage strategy. Conversely, if the total cost of direct storage is less than or equal to the total cost of incremental storage, the data update frequency is greater than its threshold, or the data access frequency is greater than its threshold, then select the direct storage strategy.

[0024] (210) The result of executing the multi-version data storage strategy. If the data version selects the direct storage strategy, the complete data version is directly stored in the database without difference or incremental processing. In the graph database, a sub-node is created on the old version data node to store the complete data of the new version. If the data version selects the incremental storage strategy, only the difference or change between adjacent versions is stored. In the graph database, an incremental attribute is added to the data node of the old version to store the difference between adjacent versions.

[0025] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the multi-version unstructured data storage method for machine learning applications.

[0026] A computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are executed by a processor to implement the multi-version unstructured data storage method for machine learning applications.

[0027] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0028] 1. This paper proposes a multi-version unstructured data storage method for machine learning applications. It can handle the multi-type, multi-version, and multi-source data storage problems in machine learning application datasets. It designs a complete data storage solution with version control, data compression, and distributed storage, achieving efficient data storage and convenient access for machine learning applications.

[0029] 2. The multi-version data storage strategy designed by the present invention comprehensively considers resource cost consumption including storage space, retrieval time, CPU resources, and network resources, as well as version updates and access frequency, and selects the optimal storage strategy for data version storage, which not only alleviates the pressure on storage space but also improves the efficiency of data storage and access.

[0030] 3. The present invention is deployed in a distributed unstructured graph database. The storage process is independent and complete, with a global perspective, which can realize the management and optimization of data storage. It is suitable for scenarios with multiple types of data sources and high-frequency data updates. The data stored in the graph database is highly scalable and can build large-scale data sets for machine learning applications.

[0031] In addition, distributed storage nodes can achieve more flexible and low-cost data storage solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the distributed storage framework of unstructured multi-source data of the present invention;

[0033] Figure 2 It is a flow chart of the multi-version data storage strategy of the present invention. DETAILED DESCRIPTION

[0034] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0035] This example provides a multi-version unstructured data storage method for machine learning applications. It stores and manages complex, multi-source data for machine learning applications in Neo4j, a graph database that supports distributed deployment. By combining various data storage costs, update frequency, and access frequency, it determines the optimal data version storage strategy and implements a neural network-based multi-type data compression algorithm to store the compressed data in a distributed graph database. This method first requires creating a distributed graph database, Neo4j, and designing a corresponding graph data model. This method then builds a distributed storage framework for unstructured multi-source data. It then uses a multi-version data storage strategy and a multi-type data compression algorithm to process and compress metadata, ultimately achieving distributed storage of unstructured data.

[0036] 1. Distributed storage framework of unstructured multi-source data

[0037] The schematic diagram of the framework is shown in Figure 1 The specific steps are as follows:

[0038] (1) Create an unstructured graph database Nep4j, and use the characteristics of Casual Clustering to configure the cluster and the client connection. Specifically, start Neo4j on the first server, initialize the cluster using the neo4j-admin command, then generate a cluster token for other servers that have started the Neo4j instance to join the cluster, perform cluster configuration, specify node roles and set cluster communication addresses, and the client can connect to the Neo4j cluster through the Bolt protocol, thereby realizing distributed deployment, and designing a corresponding graph data model, including multiple node types to represent different data entities, defining the relationship between data, in addition, each node and relationship has a series of attributes, which exist in the form of key-value pairs to store detailed information of the entity;

[0039] (2) Complex multi-source data from cameras, drones, sensors, radars, satellites, etc. will be stored in the database. Specifically, in the field of public security intelligent prevention and control, cameras obtain video streams and static images for face recognition and behavior analysis, drones collect high-resolution aerial images and videos, sensor data is diverse, including temperature, humidity and vibration information, radars provide target position and speed information, and satellite images are used for large-scale surface information collection. These multi-source data will be converted into nodes, relationships and attributes in the graph model to form a complex multi-source database;

[0040] (3) Process and judge the data through a multi-version data storage strategy, decision factors include storage space, retrieval time, CPU resource consumption, network resource consumption, and data update and access frequency, and the optimal storage strategy is decided based on the above factors and algorithm, that is, direct storage or incremental storage, and the corresponding decision result is executed;

[0041] (4) After the data is decided by the storage strategy, it will be classified according to its text, picture, video, etc. characteristics, such as based on the file extension, file header information to realize classification, then use the corresponding neural network based compression algorithm to realize data compression, specifically, text type data can use COMBINER algorithm for compression, picture type data can use TransCS algorithm for compression, and video type data can use FVC algorithm for compression;

[0042] (5) The compressed data will be combined with the decision results of the multi-version data storage strategy and stored in the distributed unstructured graph database Neo4j in the form of direct storage or incremental storage.

[0043] 2. Multi-version data storage strategy

[0044] The schematic diagram of the multi-version data storage strategy process in this embodiment is as follows Figure 2 As shown in the figure, it mainly includes data preparation, cost calculation, decision-making based on cost comparison, version frequency calculation, decision-making based on version frequency, output of decision results and execution of decision strategy. The specific steps are as follows:

[0045] (1) For multi-source data with storage requirements from sensors, cameras, drones, etc., update and record the version information of this data, including timestamps, version numbers, and other unique identification information that can record the version;

[0046] (2) Calculate the storage space cost of the version data using the direct storage strategy and the incremental storage strategy, that is, the storage space overhead required to store the version data, using C s The storage space cost of the direct storage strategy is the storage space size of the version data, as shown in formula (1), where c v is the storage space size of version v. The storage space cost of the incremental storage strategy requires calculating the storage space size of all differences in the difference set between the new and old versions, as shown in formula (2), where Δ is the difference set between versions, s e is the storage space size of the difference e. The difference set can be calculated by the difference set of the new version and the intersection of the old and new versions, as shown in formula (3), where v i is the i-th version of the dataset;

[0047] C s =c v (1)

[0048] C s =Σ e∈Δ s e (2)

[0049] Δ=v i -(v i ∩v i-1 ) (3)

[0050] (3) Calculate the retrieval time cost of the version data using the direct storage strategy and the incremental storage strategy, that is, the time cost consumed to obtain the complete data of the required version from the storage system, and use C fThe retrieval time cost of the direct storage strategy is the time consumed to find the version data, which is the time consumed by directly searching the node in the graph database, as shown in formula (4), where t v is the time it takes to find the complete data of version v, which is calculated as shown in formula (5), where ts s Is the timestamp of the read operation, ts e It is the timestamp when the data is fully loaded into the memory or retrieved by the processing unit. The retrieval of the incremental storage strategy requires first searching for the node of the previous version of the data, and then obtaining the complete data of the new version through the incremental attribute of the node of the previous version of the data. Therefore, the retrieval time cost of incremental storage is the sum of the time consumed by these two processes, as shown in formula (6), where d v is the incremental time, that is, the time to find the data of the new version v through the data of the previous version v-1. Its calculation method is shown in formula (7), where t p is the time required to parse the incremental difference data, t m is the time required to merge the incremental differences into the previous version of the data;

[0051] C f =t v (4)

[0052] t v =ts e -ts s (5)

[0053] C f =t v-1 +d v (6)

[0054] d v =t p +t m (7)

[0055] (4) Calculate the CPU resource consumption cost of the direct storage strategy and the incremental storage strategy for this version of data, that is, the CPU resources consumed by storing this version of data, using C u The CPU resource consumption cost of the direct storage strategy is the product of the time required to complete the reading, writing, copying and other operations during the data storage process and the average CPU usage rate, as shown in formula (8), where T c is the total time required to complete the operation, U u is the average CPU usage. The CPU resource consumption cost of incremental storage is the product of the time required to complete the detection, calculation, merging and other operations of incremental difference data and the average CPU usage, as shown in formula (9), where T d 、T p 、T arespectively are the time required to complete difference detection, difference calculation and difference merge operation respectively;

[0056] C u = T c × U u (8)

[0057] C u = (T d + T p + T a ) × U u (9)

[0058] (5) Calculate the network resource consumption cost of the version data using direct storage strategy and incremental storage strategy, i.e. the network bandwidth resource consumed for storing the version data, denoted as C n . The network resource consumption cost of direct storage strategy is the quotient of the complete data amount transmitted in the storage process and the network available bandwidth, as shown in formula (10), where D f is the data amount transmitted in the storage, and B n is the network available bandwidth. The network resource consumption cost of incremental storage strategy is the quotient of the difference data amount transmitted in the storage process and the network available bandwidth, as shown in formula (11), where D d is the difference data amount transmitted in the storage, which is usually less than the entire data amount;

[0059]

[0060] (6) Calculate the total cost of the version data using direct storage strategy and incremental storage strategy, denoted as C, the total cost calculation method of the two storage strategies is the weighted sum of the storage space cost, retrieval time cost, CPU resource consumption cost and network resource consumption cost, as shown in formula (12), the sum of the weight factors of the four costs is 1, and α, β, γ are the weight factors of the storage space cost, retrieval time cost and CPU resource consumption cost respectively, in the data version storage decision, the storage cost and retrieval cost are mainly considered, and the proportion of each cost in the total cost can be adjusted according to the actual situation, the direct storage total cost obtained is denoted as C d , and the incremental storage total cost is denoted as C i ;

[0061] C = αC s + βC f + γC u + (1-α-β-γ)C n (12)

[0062] (7) Calculate the version update frequency, i.e. the frequency of each version data being updated, denoted as F uvIt is calculated by dividing the number of times the data is updated within a certain time interval by the time difference, as shown in formula (13), where n v is the number of times version v is updated in the time interval [t1, t2]. A version update frequency threshold is set according to the actual situation, and F is used to calculate the frequency of the update. ut express;

[0063]

[0064] (8) Calculate the version access frequency, that is, the frequency at which each version data is accessed, using F av It is calculated by dividing the number of times the data is accessed within a certain time interval by the time difference, as shown in formula (14), where a v is the number of times version v is accessed in the time interval [t1, t2]. A version access frequency threshold is set according to the actual situation, and F is used to calculate the access frequency of the version v. at express;

[0065]

[0066] (9) Comprehensively consider factors such as the total cost of direct storage strategy and incremental storage strategy, version update frequency, version access frequency, etc., make a decision on the data version storage strategy, and select the optimal storage method. Specifically, if the total cost of direct storage is greater than the total cost of incremental storage, the data update frequency is less than its threshold, and the data access frequency is also less than its threshold, that is, C d >C i 、F uv <F ut And F av <F at , then the incremental storage strategy is selected. On the contrary, if the total cost of direct storage is less than or equal to the total cost of incremental storage, the data update frequency is greater than or equal to its threshold, or the data access frequency is greater than or equal to its threshold, that is, C d ≤C i 、F av ≥F at or F av ≥F at , then select the direct storage strategy;

[0067] (10) The result of executing the multi-version data storage strategy is that if the data version selects the direct storage strategy, the complete data version is directly stored in the database without performing differential or incremental processing. In the graph database Neo4j, a child node is created on the old version data node to store the complete data of the new version. If the data version selects the incremental storage strategy, only the differences or changes between adjacent versions are stored. In the graph database Neo4j, an incremental attribute is added to the old version data node to store the differences between adjacent versions.

[0068] The above disclosure is only a preferred embodiment of the present invention and cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A multi-version unstructured data storage method for machine learning applications, characterized by: The method comprises: (11) Create a distributed unstructured graph database and design the corresponding graph data model, including graph nodes, data relationships, attributes and their corresponding representations; (12) For multi-source data to be stored in the database, the data is processed and judged through a multi-version data storage strategy, and the optimal storage strategy is selected based on storage space, retrieval time, CPU resource consumption cost, network resource consumption cost, and data update and access frequency, i.e., direct storage or incremental storage; (13) After the strategic decision is made, the data will be classified according to the characteristics of its text, image, and video types, and then the corresponding neural network-based compression algorithm will be used to compress the data of each modality; (14) The compressed data will be stored in a distributed unstructured graph database in the form of direct storage or incremental storage; The step (12) specifically includes: (21) For multi-source data with storage requirements from sensors, cameras, and drones, update and record the version information of these data; (22) Calculate the storage space cost of the version data using direct storage and incremental storage. The storage space cost is the storage space overhead required to store the version data. The storage space cost of the direct storage strategy is the storage space size of the version data. The storage space cost of the incremental storage strategy requires calculating the storage space size of all differences in the difference set between the new and old versions. The difference set is obtained by performing a difference set calculation on the intersection of the new version and the old and new versions. (23) Calculate the retrieval time cost of the version data using direct storage and incremental storage. The retrieval time cost is the time cost of obtaining the complete data of the required version from the storage system. The retrieval time cost of the direct storage strategy is the time consumed to find the version data, which is the time consumed by directly searching the node in the graph database. The retrieval of the incremental storage strategy requires first searching the node of the previous version data, and then obtaining the complete data of the new version through the incremental attributes of the previous version data node. Therefore, the retrieval time cost of the incremental storage is the sum of the time consumed by these two processes; (24) Calculate the CPU resource consumption cost of direct storage and incremental storage for this version of data. The CPU resource consumption cost is the CPU resource consumed by storing this version of data. The CPU resource consumption cost of the direct storage strategy is the product of the time required to complete the read, write, and copy operations during data storage and the average CPU usage rate. The CPU resource consumption cost of incremental storage is the product of the time required to complete the detection, calculation, and merging operations of incremental difference data and the average CPU usage rate. (25) Calculate the network resource consumption cost of direct storage and incremental storage for this version of data. The network resource consumption cost is the network bandwidth consumed by storing this version of data. The network resource consumption cost of the direct storage strategy is the quotient of the amount of complete data transmitted during the storage process and the available network bandwidth. The network resource consumption cost of the incremental storage strategy is the quotient of the amount of differential data transmitted during the storage process and the available network bandwidth. (26) Calculate the total cost of direct storage and incremental storage for this version of data. The total cost of the two storage strategies is calculated as the weighted sum of storage space cost, retrieval time cost, CPU resource consumption cost, and network resource consumption cost. The sum of the weight factors of the four costs is 1. (27) Calculate the version update frequency, that is, the frequency at which each version data is updated. The calculation method is to take the quotient of the number of times the data is updated in a certain time interval and the time difference, and set a version update frequency threshold; (28) Calculate the version access frequency, that is, the frequency with which each version of data is accessed. The calculation method is to divide the number of times the data is accessed within a certain time interval by the time difference, and set a version access frequency threshold; (29) Comprehensively consider the total cost of direct storage and incremental storage, version update frequency, and version access frequency factors, make a decision on the data version storage strategy, and select the optimal storage method. Specifically, if the total cost of direct storage is greater than the total cost of incremental storage, the data update frequency is less than its threshold, and the data access frequency is also less than its threshold, then the incremental storage strategy is selected. Conversely, if the total cost of direct storage is less than or equal to the total cost of incremental storage, the data update frequency is greater than its threshold, or the data access frequency is greater than its threshold, then the direct storage strategy is selected. (210) The result of executing the multi-version data storage strategy is that if the data version selects the direct storage strategy, the complete data version is directly stored in the database without performing difference or incremental processing. In the graph database, a child node is created on the old version data node to store the complete data of the new version. If the data version selects the incremental storage strategy, only the differences or changes between adjacent versions are stored. In the graph database, an incremental attribute is added to the old version data node to store the differences between adjacent versions.

2. The multi-version unstructured data storage method for machine learning applications according to claim 1, characterized in that: Said step (12) further comprises: (31) For multi-source data with storage requirements from sensors, cameras, and drones, update and record the version information of these data, including timestamps and version numbers to record the unique identification information of the version; (32) Calculate the storage space cost of the version data using the direct storage strategy and the incremental storage strategy, that is, the storage space cost required to store the version data, using The storage space cost of the direct storage strategy is the storage space size of the version data, as shown in formula (1), where Is the version The storage space size of the incremental storage strategy requires calculating the storage space size of all differences in the difference set between the new and old versions, as shown in formula (2), where is the set of differences between versions, It's the difference The storage space size of the new version is calculated by the difference set of the intersection of the new and old versions, as shown in formula (3), where It is versions of the dataset; (33) Calculate the retrieval time cost of the version data using the direct storage strategy and the incremental storage strategy, that is, the time cost consumed to obtain the complete data of the required version from the storage system, using The retrieval time cost of the direct storage strategy is the time consumed to find the version data, which is the time consumed by directly searching the node in the graph database, as shown in formula (4), where The version is found The time consumed by the complete data is calculated as shown in formula (5), where is the timestamp of when the read operation was initiated, It is the timestamp when the data is fully loaded into the memory or retrieved by the processing unit. The retrieval of the incremental storage strategy needs to first search for the node of the previous version of the data, and then obtain the complete data of the new version through the incremental attributes of the previous version of the data node. Therefore, the retrieval time cost of incremental storage is the sum of the time consumed by these two processes, as shown in formula (6), where is the incremental time, that is, the time from the previous version A new version of the data was found The time of the data is calculated as shown in formula (7), where is the time required to parse the incremental difference data, is the time required to merge the incremental differences into the previous version of the data; (34) Calculate the CPU resource consumption cost of the direct storage strategy and the incremental storage strategy for this version of data, that is, the CPU resources consumed by storing this version of data, using It means that the CPU resource consumption cost of the direct storage strategy is the product of the time required to complete the read, write, and copy operations during data storage and the average CPU usage, as shown in formula (8), where is the total time required to complete the operation, is the average CPU usage rate. The CPU resource consumption cost of incremental storage is the product of the time required to complete the detection, calculation, and merging operations of incremental difference data and the average CPU usage rate, as shown in formula (9), where 、 、 The time required to complete the difference detection, difference calculation and difference merging operations respectively; (35) Calculate the network resource consumption cost of the direct storage strategy and the incremental storage strategy for this version of data, that is, the network bandwidth resources consumed by storing this version of data, using The network resource consumption cost of the direct storage strategy is the quotient of the complete data volume transmitted during the storage process and the available network bandwidth, as shown in formula (10), where is the amount of data stored and transmitted, is the available network bandwidth. The network resource consumption cost of the incremental storage strategy is the quotient of the differential data volume transmitted during the storage process and the available network bandwidth, as shown in formula (11), where It is the difference in the amount of data stored and transmitted, which is usually smaller than the total amount of data; (36) Calculate the total cost of using direct storage strategy and incremental storage strategy for this version of data, using It means that the total cost of the two storage strategies is calculated as the weighted sum of storage space cost, retrieval time cost, CPU resource consumption cost, and network resource consumption cost. As shown in formula (12), the sum of the weight factors of the four costs is 1. 、 、 are the weight factors of storage space cost, retrieval time cost, and CPU resource consumption cost respectively. The total direct storage cost is calculated using Indicates that the total incremental storage cost is express; (37) Calculate the version update frequency, that is, the frequency at which each version data is updated, using It is calculated by dividing the number of times the data is updated within a certain time interval by the time difference, as shown in formula (13), where Is the version In the time interval The number of times it has been updated in the past, set a version update frequency threshold, and use express; (38) Calculate the version access frequency, that is, the frequency with which each version data is accessed, using It is calculated by dividing the number of times the data is accessed within a certain time interval by the time difference, as shown in formula (14), where Is the version In the time interval The number of times it is accessed within a certain period, set a version access frequency threshold, and use express; (39) Comprehensively consider the total cost, version update frequency, and version access frequency of the direct storage strategy and the incremental storage strategy, make a decision on the data version storage strategy, and select the optimal storage method. Specifically, if the total cost of direct storage is greater than the total cost of incremental storage, the data update frequency is less than its threshold, and the data access frequency is also less than its threshold, that is, 、 and , then the incremental storage strategy is selected. On the contrary, if the total cost of direct storage is less than or equal to the total cost of incremental storage, the data update frequency is greater than or equal to its threshold, or the data access frequency is greater than or equal to its threshold, that is, 、 or , then select the direct storage strategy; (310) The result of executing the multi-version data storage strategy is that if the data version selects the direct storage strategy, the complete data version will be directly stored in the database without performing differential or incremental processing. In the graph database Neo4j, a child node will be created on the old version data node to store the complete data of the new version. If the data version selects the incremental storage strategy, only the differences or changes between adjacent versions will be stored. In the graph database Neo4j, an incremental attribute will be added to the old version data node to store the differences between adjacent versions.

3. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the multi-version unstructured data storage method for machine learning applications as described in any one of claims 1 to 2 above is implemented.

4. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the multi-version unstructured data storage method for machine learning applications as described in any one of claims 1-2 is implemented.

Images