Multi-version unstructured data storage method for machine learning application

Abstract

The invention discloses a machine learning application-oriented multi-version unstructured data storage method, which specifically comprises the following steps of: (1) a distributed storage framework of unstructured multi-source data: processing and judging the multi-source data by utilizing a multi-version data storage strategy before the data is formally stored; the optimal storage strategy is selected based on the balance of the storage space and the retrieval time, and the data compression technology based on the neural network correspondingly compresses different types of data, so that the storage efficiency of a distributed database and the convenience of machine learning application in the data access and processing process are improved. (2) a multi-version data storage strategy: facing challenges of continuous generation of multi-source data and continuous change of storage versions; the method can be deployed in a distributed unstructured graph database, the storage process is independent and complete and has a global perspective, management and optimization of data storage can be realized, and the method is suitable for scenes of multi-type data sources and high-frequency data updating.

Description

Technical Field

[0001] The present invention relates to an unstructured data storage method, and in particular to a multi-type and multi-version data storage method for machine learning applications. Background Art

[0002] In the field of machine learning, data sets are the core of model training and evaluation. With the advancement of technology and the expansion of applications, the types of data sets are becoming more and more diverse. For example, in the field of public security intelligent prevention and control, multi-type data such as text, pictures, and videos from sensors, cameras, satellites, and drones support the supervision analysis and decision-making governance of relevant departments. These data are not only huge in volume, but also have different structures, becoming an important resource for machine learning applications. However, different types of data require different storage formats and processing methods. Traditional data compression schemes are difficult to meet the requirements of real-time and convenience of machine learning data processing. At the same time, the continuously generated data also generates multiple versions of data sets that need to be stored at all times. Directly storing each version will not only greatly increase the storage cost, but also bring great challenges to retrieval and implementation. In practical applications, data version management not only needs to support incremental storage, but also needs to retrieve and compare between multiple versions. Therefore, how to efficiently store and access these multi-version unstructured data is one of the technical challenges facing current data storage.

[0003] Currently, data storage solutions for machine learning applications mainly focus on the optimization of storage architecture. Distributed file systems (such as HDFS) and object storage systems (such as Amazon S3) are widely used for large-scale data storage to achieve efficient data management and access. However, these storage solutions still face many challenges when processing unstructured data. Traditional centralized relational databases are difficult to meet the storage requirements of large-scale unstructured data, especially in high-concurrency and high-throughput scenarios. In addition, comprehensive storage solutions for multi-type data are not yet fully mature and are not easy to deploy widely in actual environments. Therefore, a distributed data storage framework that can meet the needs of multi-type data storage and frequent version updates is needed.

[0004] Faced with the complex multi-source data storage scenarios of machine learning applications, the massive data of multiple types and versions has brought about the challenges of sharply rising storage costs and increased complexity of data retrieval and version management. This method proposes a multi-version unstructured data storage solution for machine learning applications. Through the multi-version data storage strategy, the optimal storage strategy is adopted for multi-version metadata, and different types of metadata are compressed based on the neural network algorithm. The compressed data is stored in a distributed unstructured graph database, providing an efficient and low-cost unstructured data storage solution for machine learning applications. Summary of the invention

[0005] Purpose of the invention: The present invention is aimed at the problem of multi-type and multi-version data storage in machine learning application scenarios, and provides an unstructured distributed data storage method in a distributed database, including a distributed storage framework for unstructured multi-source data and a multi-version data storage strategy. Specifically, when multi-source data from sensors, cameras, and drones need to be stored, this method does not store the data in a database through a simple method, but adopts a multi-version data storage strategy to select the optimal storage strategy for the version of the data based on the trade-off between storage space and retrieval time, that is, direct storage or incremental storage. After the policy decision, the metadata will be compressed using the corresponding neural network algorithm according to different types such as text, pictures, and videos to improve storage efficiency. The compressed data is stored in a distributed unstructured database to form a data set for convenient access and management for machine learning.

[0006] Technical solution: The multi-version unstructured data storage method for machine learning applications of the present invention includes:

[0007] (1) Distributed storage framework for unstructured multi-source data: The data storage framework is aimed at multi-source data to be stored in the database. In view of its complex types and frequent version updates, it implements a complete data storage process of version control, classification compression, and distributed storage. Specifically, it includes the implementation of multi-version data storage strategies, multi-type data compression algorithms, and distributed graph database storage solutions.

[0008] (2) Multi-version data storage strategy: Since the multi-source data from sensors, cameras, and drones is huge and updated frequently, the multi-version data storage strategy comprehensively considers the storage space cost, retrieval time cost, CPU resource consumption cost, and network resource consumption cost, and combines the frequency of version updates and access to select the optimal storage strategy for different data versions, namely direct storage or incremental storage.

[0009] Furthermore, the distributed storage framework of unstructured multi-source data in step (1) specifically includes:

[0010] (11) Create a distributed unstructured graph database and design the corresponding graph data model, including graph nodes, data relationships, attributes and their corresponding representations;

[0011] (12) For multi-source data to be stored in the database, the data is processed and judged through a multi-version data storage strategy, and the optimal storage strategy is selected based on the costs such as storage space and retrieval time and the frequency of data update and access, i.e., direct storage or incremental storage;

[0012] (13) After the strategic decision is made, the data will be classified according to its characteristics such as text, picture, video, etc., and then the corresponding neural network-based compression algorithm will be used to achieve data compression;

[0013] (14) The compressed data will be stored in a distributed unstructured graph database.

[0014] Furthermore, the multi-version data storage strategy in step (2) specifically includes:

[0015] (21) Update and record the version information of multi-source data with storage requirements from sensors, cameras, and drones;

[0016] (22) Calculate the storage space cost of the version data using direct storage and incremental storage. The storage space cost is the storage space overhead required to store the version data. The storage space cost of the direct storage strategy is the storage space size of the version data. The storage space cost of the incremental storage strategy requires calculating the storage space size of all differences in the difference set between the new and old versions. The difference set can be obtained by performing a difference set calculation on the intersection of the new version and the new and old versions.

[0017] (23) Calculate the retrieval time cost of the version data using direct storage and incremental storage. The retrieval time cost is the time cost of obtaining the required version of complete data from the storage system. The retrieval time cost of the direct storage strategy is the time consumed to find the version data, which is the time consumed by directly searching the node in the graph database. The retrieval of the incremental storage strategy requires first searching the node of the previous version data, and then obtaining the complete data of the new version through the incremental attributes of the previous version data node. Therefore, the retrieval time cost of the incremental storage is the sum of the time consumed by these two processes;

[0018] (24) Calculate the CPU resource consumption cost of direct storage and incremental storage for this version of data. The CPU resource consumption cost is the CPU resource consumed by storing this version of data. The CPU resource consumption cost of the direct storage strategy is the product of the time required to complete operations such as reading, writing, and copying during data storage and the average CPU usage rate. The CPU resource consumption cost of incremental storage is the product of the time required to complete operations such as detection, calculation, and merging of incremental difference data and the average CPU usage rate.

[0019] (25) Calculate the network resource consumption cost of the direct storage and incremental storage of the version data. The network resource consumption cost is the network bandwidth resource consumed by storing the version data. The network resource consumption cost of the direct storage strategy is the quotient of the complete data volume transmitted during the storage process and the available network bandwidth. The network resource consumption cost of the incremental storage strategy is the quotient of the differential data volume transmitted during the storage process and the available network bandwidth.

[0020] (26) Calculate the total cost of direct storage and incremental storage for the version data. The total cost is represented by C. The total cost of the two storage strategies is calculated as the weighted sum of storage space cost, retrieval time cost, CPU resource consumption cost, and network resource consumption cost. The sum of the weight factors of the four costs is 1. In the data version storage decision, the storage cost and retrieval cost are mainly used. The proportion of each cost in the total cost can be adjusted according to the actual situation.

[0021] (27) Calculate the version update frequency, that is, the frequency at which each version of data is updated. The calculation method is the quotient of the number of times the data is updated within a certain time interval and the time difference. In addition, set a version update frequency threshold based on actual conditions;

[0022] (28) Calculate the version access frequency, that is, the frequency with which each version of data is accessed. The calculation method is the quotient of the number of times the data is accessed within a certain time interval and the time difference. In addition, set a version access frequency threshold based on actual conditions.

[0023] (29) Comprehensively consider factors such as the total cost of direct storage and incremental storage, version update frequency, version access frequency, etc., make a decision on the data version storage strategy, and select the optimal storage method. Specifically, if the total cost of direct storage is greater than the total cost of incremental storage, the data update frequency is less than its threshold, and the data access frequency is also less than its threshold, then the incremental storage strategy is selected. On the contrary, if the total cost of direct storage is less than or equal to the total cost of incremental storage, the data update frequency is greater than its threshold, or the data access frequency is greater than its threshold, then the direct storage strategy is selected.

[0024] (210) The result of executing the multi-version data storage strategy is that if the data version selects the direct storage strategy, the complete data version is directly stored in the database without performing difference or incremental processing. In the graph database, a child node is created on the old version data node to store the complete data of the new version. If the data version selects the incremental storage strategy, only the differences or changes between adjacent versions are stored. In the graph database, an incremental attribute is added to the old version data node to store the differences between adjacent versions.

[0025] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the multi-version unstructured data storage method for machine learning applications when executing the program.

[0026] A computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the multi-version unstructured data storage method for machine learning applications.

[0027] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0028] 1. The present invention proposes a multi-version unstructured data storage method for machine learning applications, which can handle the multi-type, multi-version multi-source data storage problem in machine learning application data sets, designs a complete data storage solution with version control, data compression, and distributed storage, and realizes efficient data storage and convenient access to machine learning applications.

[0029] 2. The multi-version data storage strategy designed by the present invention comprehensively considers resource cost consumption including storage space, retrieval time, CPU resources, and network resources, as well as version updates and access frequencies, and selects the optimal storage strategy for data version storage, which not only alleviates the pressure on storage space, but also improves the efficiency of data storage and access.

[0030] 3. The present invention is deployed in a distributed unstructured graph database. The storage process is independent and complete, with a global perspective. It can realize the management and optimization of data storage and is suitable for scenarios with multiple types of data sources and high-frequency data updates. The data stored in the graph database has strong scalability and can build large-scale data sets for machine learning applications.

[0031] In addition, distributed storage nodes can achieve more flexible and low-cost data storage solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of a distributed storage framework of unstructured multi-source data of the present invention;

[0033] Figure 2 It is a schematic diagram of the multi-version data storage strategy process of the present invention. DETAILED DESCRIPTION

[0034] The specific implementation modes of the present invention are described in detail below with reference to the accompanying drawings.

[0035] This example provides a multi-version unstructured data storage method for machine learning applications. Multi-source complex data for machine learning applications is stored and managed in Neo4j, a graph database that supports distributed deployment. The optimal strategy for data version storage is determined by combining various data storage costs, update frequency, and access frequency. A multi-type data compression algorithm based on a neural network is implemented to store the compressed data in a distributed graph database. This method first requires the creation of a distributed graph database Neo4j, the design of a corresponding graph data model, the construction of a distributed storage framework for unstructured multi-source data, and then the processing and compression of metadata through a multi-version data storage strategy and a multi-type data compression algorithm, ultimately achieving distributed storage of unstructured data.

[0036] 1. Distributed storage framework for unstructured multi-source data

[0037] The schematic diagram of this framework is as follows Figure 1 As shown, the specific steps are as follows:

[0038] (1) Create an unstructured graph database Nep4j and use its causal clustering feature to configure the cluster and client connection. Specifically, start Neo4j on the first server and use the neo4j-admin command to initialize the cluster. Then generate a cluster token for other servers that have started Neo4j instances to join the cluster. Perform cluster configuration, specify node roles, and set cluster communication addresses. Clients can connect to the Neo4j cluster through the Bolt protocol to achieve distributed deployment. Design a corresponding graph data model, including multiple node types to represent different data entities and define the relationship between data. In addition, each node and relationship has a series of attributes in the form of key-value pairs to store detailed information about the entity.

[0039] (2) Complex multi-source data from cameras, drones, sensors, radars, satellites, etc. will be stored in the database. Specifically, in the field of public safety intelligent prevention and control, cameras obtain video streams and static images for face recognition and behavior analysis, drones collect high-resolution aerial images and videos, and sensor data is diverse, including temperature, humidity, and vibration information. Radars provide location and speed information of targets, and satellite images are used to collect large-scale surface information. These multi-source data will be converted into nodes, relationships, and attributes in the graph model to form a complex multi-source database;

[0040] (3) Process and judge the data through a multi-version data storage strategy. The decision factors include storage space, retrieval time, CPU resource consumption, network resource consumption and other costs, as well as the frequency of data update and access. Based on the above factors, the algorithm determines the optimal storage strategy, i.e., direct storage or incremental storage, and executes the corresponding decision result.

[0041] (4) After the storage strategy decision is made, the data will be classified according to its characteristics such as text, image, video, etc., such as classification based on the file extension and file header information of the data, and then the corresponding neural network-based compression algorithm will be used to achieve data compression. Specifically, text type data can be compressed using the COMBINER algorithm, image type data can be compressed using the TransCS algorithm, and video type data can be compressed using the FVC algorithm;

[0042] (5) The compressed data will be combined with the decision results of the multi-version data storage strategy and stored in the distributed unstructured graph database Neo4j in the form of direct storage or incremental storage.

[0043] 2. Multi-version data storage strategy

[0044] The schematic diagram of the multi-version data storage strategy process in this embodiment is as follows Figure 2 As shown in the figure, it mainly includes data preparation, cost calculation, decision-making based on cost comparison, version frequency calculation, decision-making based on version frequency, output of decision results and execution of decision strategies, etc. The specific steps are as follows:

[0045] (1) For multi-source data with storage requirements from sensors, cameras, drones, etc., update and record the version information of these data, including timestamps, version numbers, and other unique identification information that can record the version;

[0046] (2) Calculate the storage space cost of the version data using the direct storage strategy and the incremental storage strategy, that is, the storage space cost required to store the version data, using C s The storage space cost of the direct storage strategy is the storage space size of the version data, as shown in formula (1), where c v is the storage space size of version v. The storage space cost of the incremental storage strategy needs to calculate the storage space size of all differences in the difference set between the new and old versions, as shown in formula (2), where Δ is the difference set between versions, s e is the storage space size of the difference e. The difference set can be obtained by calculating the difference set of the new version and the intersection of the new and old versions, as shown in formula (3), where v i is the i-th version of the dataset;

[0047] C s =c v (1)

[0048] C s =Σ e∈Δ s e (2)

[0049] Δ=v i -(v i ∩v i-1 ) (3)

[0050] (3) Calculate the retrieval time cost of the version data using the direct storage strategy and the incremental storage strategy, that is, the time cost consumed to obtain the complete data of the required version from the storage system, and use C fThe retrieval time cost of the direct storage strategy is the time consumed to find the version data, which is the time consumed to directly search for nodes in the graph database, as shown in formula (4), where t v is the time it takes to find the complete data of version v, which is calculated as shown in formula (5), where ts s is the timestamp of the read operation, ts e It is the timestamp when the data is fully loaded into the memory or retrieved by the processing unit. The retrieval of the incremental storage strategy needs to first search for the node of the previous version of the data, and then obtain the complete data of the new version through the incremental attributes of the previous version of the data node. Therefore, the retrieval time cost of incremental storage is the sum of the time consumed by these two processes, as shown in formula (6), where d v is the incremental time, that is, the time to find the data of the new version v through the data of the previous version v-1. Its calculation method is shown in formula (7), where t p is the time required to parse the incremental difference data, t m is the time required to merge the incremental differences into the previous version of the data;

[0051] C f =t v (4)

[0052] t v =ts e -ts s (5)

[0053] C f =t v-1 +d v (6)

[0054] d v =t p +t m (7)

[0055] (4) Calculate the CPU resource consumption cost of the direct storage strategy and the incremental storage strategy for the version data, that is, the CPU resources consumed by storing the version data, using C u The CPU resource consumption cost of the direct storage strategy is the product of the time required to complete operations such as reading, writing, and copying during data storage and the average CPU usage rate, as shown in formula (8), where T c is the total time required to complete the operation, U u is the average CPU usage. The CPU resource consumption cost of incremental storage is the product of the time required to complete the detection, calculation, merging and other operations of incremental difference data and the average CPU usage, as shown in formula (9), where T d , T p , T aThey are the time required to complete the difference detection, difference calculation and difference merging operations respectively;

[0056] C u =T c ×U u (8)

[0057] C u =(T d +T p +T a )×U u (9)

[0058] (5) Calculate the network resource consumption cost of the direct storage strategy and the incremental storage strategy for this version of data, that is, the network bandwidth resources consumed by storing this version of data, using C n The network resource consumption cost of the direct storage strategy is the quotient of the complete data volume transmitted during the storage process and the available network bandwidth, as shown in formula (10), where D f is the amount of data stored and transmitted, B n is the available network bandwidth. The network resource consumption cost of the incremental storage strategy is the quotient of the differential data volume transmitted during the storage process and the available network bandwidth, as shown in formula (11), where D d It is the difference in the amount of data stored and transmitted, which is usually smaller than the entire amount of data;

[0059]

[0060] (6) Calculate the total cost of the direct storage strategy and the incremental storage strategy for the version data, denoted by C. The total cost of the two storage strategies is calculated as the weighted sum of storage space cost, retrieval time cost, CPU resource consumption cost, and network resource consumption cost, as shown in formula (12). The sum of the weight factors of the four costs is 1. α, β, and γ are the weight factors of storage space cost, retrieval time cost, and CPU resource consumption cost, respectively. In the data version storage decision, storage cost and retrieval cost are mainly used. The proportion of each cost in the total cost can be adjusted according to the actual situation. The calculated total cost of direct storage is denoted by C. d The total incremental storage cost is represented by C i express;

[0061] C=αC s +βC f +γC u +(1-α-β-γ)C n (12)

[0062] (7) Calculate the version update frequency, that is, the frequency at which each version data is updated, using F uvIt is calculated by the quotient of the number of times the data is updated in a certain time interval and the time difference, as shown in formula (13), where n v is the number of times version v is updated in the time interval [t1, t2]. A version update frequency threshold is set according to the actual situation, and F is used to represent the frequency of version update. ut express;

[0063]

[0064] (8) Calculate the version access frequency, that is, the frequency with which each version data is accessed, using F av It is calculated by the quotient of the number of times the data is accessed in a certain time interval and the time difference, as shown in formula (14), where a v is the number of times version v is accessed in the time interval [t1, t2]. A version access frequency threshold is set according to the actual situation, and F is used to represent the access frequency of the version v. at express;

[0065]

[0066] (9) Comprehensively consider the total cost, version update frequency, version access frequency and other factors of the direct storage strategy and the incremental storage strategy, make a decision on the data version storage strategy, and select the optimal storage method. Specifically, if the total cost of direct storage is greater than the total cost of incremental storage, the data update frequency is less than its threshold, and the data access frequency is also less than its threshold, that is, C d >C i 、F uv <F ut And F av <F at , then the incremental storage strategy is selected. On the contrary, if the total cost of direct storage is less than or equal to the total cost of incremental storage, the data update frequency is greater than or equal to its threshold, or the data access frequency is greater than or equal to its threshold, that is, C d ≤C i 、F av ≥F at or F av ≥F at , then select the direct storage strategy;

[0067] (10) The result of executing the multi-version data storage strategy is that if the data version selects the direct storage strategy, the complete data version is directly stored in the database without performing difference or incremental processing. In the graph database Neo4j, a child node is created on the old version data node to store the complete data of the new version. If the data version selects the incremental storage strategy, only the differences or changes between adjacent versions are stored. In the graph database Neo4j, an incremental attribute is added to the old version data node to store the differences between adjacent versions.

[0068] The above disclosure is only a preferred embodiment of the present invention, and cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A multi-version unstructured data storage method for machine learning applications, characterized in that: The method comprises: (1) Distributed storage framework for unstructured multi-source data. The data storage framework is aimed at multi-source data to be stored in the database. In view of its complex types and frequent version updates, it implements a complete data storage process of version control, classification compression, and distributed storage. Specifically, it implements version control through a multi-version data storage strategy. After decision-making, the data is classified according to its type characteristics, and then the corresponding neural network-based compression algorithm is used to implement data compression. The compressed data is stored in a distributed graph database. (2) Multi-version data storage strategy. The multi-version data storage strategy comprehensively considers the storage space cost, retrieval time cost, CPU resource consumption cost, network resource consumption cost, and combines the frequency of version updates and access to select the optimal storage strategy for different data versions, namely direct storage or incremental storage.

2. The multi-version unstructured data storage method for machine learning applications according to claim 1, characterized in that: The distributed storage framework of unstructured multi-source data in step (1) specifically includes: (11) Create a distributed unstructured graph database and design the corresponding graph data model, including graph nodes, data relationships, attributes and their corresponding representations; (12) For multi-source data to be stored in the database, the data is processed and judged through a multi-version data storage strategy, and the optimal storage strategy is selected based on the costs of storage space, retrieval time, CPU resource consumption cost, network resource consumption, and the frequency of data update and access, i.e., direct storage or incremental storage; (13) After the strategic decision is made, the data will be classified according to its characteristics such as text, picture, video, etc., and then the corresponding neural network-based compression algorithm will be used to compress the data of each modality; (14) The compressed data will be stored in a distributed unstructured graph database in the form of direct storage or incremental storage.

3. The multi-version unstructured data storage method for machine learning applications according to claim 1, characterized in that: The step (2) specifically comprises: (21) Update and record the version information of multi-source data with storage requirements from sensors, cameras, and drones; (22) Calculate the storage space cost of the version data using direct storage and incremental storage. The storage space cost is the overhead of the storage space required to store the version data. The storage space cost of the direct storage strategy is the storage space size of the version data. The storage space cost of the incremental storage strategy requires calculating the storage space size of all differences in the difference set between the new and old versions, where the difference set is obtained by performing a difference set calculation on the intersection of the new version and the new and old versions. (23) Calculate the retrieval time cost of the version data using direct storage and incremental storage. The retrieval time cost is the time cost of obtaining the required version of complete data from the storage system. The retrieval time cost of the direct storage strategy is the time consumed to find the version data, which is the time consumed by directly searching the node in the graph database. The retrieval of the incremental storage strategy requires first searching the node of the previous version data, and then obtaining the complete data of the new version through the incremental attributes of the previous version data node. Therefore, the retrieval time cost of the incremental storage is the sum of the time consumed by these two processes. (24) Calculate the CPU resource consumption cost of using direct storage and incremental storage for the version of data. The CPU resource consumption cost is the CPU resource consumed by storing the version of data. The CPU resource consumption cost of the direct storage strategy is the product of the time required to complete operations such as reading, writing, and copying during data storage and the average CPU usage rate. The CPU resource consumption cost of the incremental storage strategy is the product of the time required to complete operations such as detection, calculation, and merging of incremental difference data and the average CPU usage rate. (25) Calculate the network resource consumption cost of the direct storage and incremental storage of the version data. The network resource consumption cost is the network bandwidth resources consumed by storing the version data. The network resource consumption cost of the direct storage strategy is the quotient of the complete data volume transmitted during the storage process and the available network bandwidth. The network resource consumption cost of the incremental storage strategy is the quotient of the differential data volume transmitted during the storage process and the available network bandwidth. (26) Calculate the total cost of direct storage and incremental storage for the version data. The total cost of the two storage strategies is calculated as the weighted sum of storage space cost, retrieval time cost, CPU resource consumption cost, and network resource consumption cost. The sum of the weight factors of the four costs is 1. In the data version storage decision, storage cost and retrieval cost are mainly used. The proportion of each cost in the total cost can be adjusted according to the actual situation. (27) Calculate the version update frequency, that is, the frequency at which each version of data is updated. The calculation method is the quotient of the number of times the data is updated within a certain time interval and the time difference. In addition, set a version update frequency threshold based on actual conditions; (28) Calculate the version access frequency, that is, the frequency with which each version of data is accessed. The calculation method is the quotient of the number of times the data is accessed within a certain time interval and the time difference. In addition, set a version access frequency threshold based on actual conditions. (29) Comprehensively consider factors such as the total cost of direct storage and incremental storage, version update frequency, version access frequency, etc., make a decision on the data version storage strategy, and select the optimal storage method. Specifically, if the total cost of direct storage is greater than the total cost of incremental storage, the data update frequency is less than its threshold, and the data access frequency is also less than its threshold, then the incremental storage strategy is selected. On the contrary, if the total cost of direct storage is less than or equal to the total cost of incremental storage, the data update frequency is greater than its threshold, or the data access frequency is greater than its threshold, then the direct storage strategy is selected. (210) The result of executing the multi-version data storage strategy is that if the data version selects the direct storage strategy, the complete data version is directly stored in the database without performing difference or incremental processing. In the graph database, a child node is created on the old version data node to store the complete data of the new version. If the data version selects the incremental storage strategy, only the differences or changes between adjacent versions are stored. In the graph database, an incremental attribute is added to the old version data node to store the differences between adjacent versions.

4. The multi-version unstructured data storage method for machine learning applications according to claim 1, characterized in that: The step (2) specifically comprises: (21) For multi-source data with storage requirements from sensors, cameras, drones, etc., update and record the version information of these data, including timestamps, version numbers, and unique identification information of the versions; (22) Calculate the storage space cost of the version data using the direct storage strategy and the incremental storage strategy, that is, the storage space cost required to store the version data, using C s The storage space cost of the direct storage strategy is the storage space size of the version data, as shown in formula (1), where c v is the storage space size of version v. The storage space cost of the incremental storage strategy needs to calculate the storage space size of all differences in the difference set between the new and old versions, as shown in formula (2), where Δ is the difference set between versions, s e is the storage space size of the difference e. The difference set is obtained by calculating the difference set of the new version and the intersection of the new and old versions, as shown in formula (3), where v i is the i-th version of the dataset; C s =c v (1) C s =∑ e∈Δ s e (2) Δ=v i -(v i ∩v i-1 ) (3) (23) Calculate the retrieval time cost of the version data using the direct storage strategy and the incremental storage strategy, that is, the time cost consumed to obtain the complete data of the required version from the storage system, and use C f The retrieval time cost of the direct storage strategy is the time consumed to find the version data, which is the time consumed to directly search for nodes in the graph database, as shown in formula (4), where t v is the time it takes to find the complete data of version v, which is calculated as shown in formula (5), where ts s is the timestamp of the read operation, ts e It is the timestamp when the data is fully loaded into the memory or retrieved by the processing unit. The retrieval of the incremental storage strategy needs to first search for the node of the previous version of the data, and then obtain the complete data of the new version through the incremental attribute of the previous version of the data node. Therefore, the retrieval time cost of incremental storage is the sum of the time consumed by these two processes, as shown in formula (6), where d v is the incremental time, that is, the time to find the data of the new version v through the data of the previous version v-1. Its calculation method is shown in formula (7), where t p is the time required to parse the incremental difference data, t m is the time required to merge the incremental differences into the previous version of the data; C f =t v (4) t v =ts e -ts s (5) C f =t v-1 +d v (6) d v =t p +t m (7) (24) Calculate the CPU resource consumption cost of the direct storage strategy and the incremental storage strategy for this version of data, that is, the CPU resources consumed by storing this version of data, using C u It means that the CPU resource consumption cost of the direct storage strategy is the product of the time required to complete the read, write, and copy operations during data storage and the average CPU usage rate, as shown in formula (8), where T c is the total time required to complete the operation, U u is the average CPU usage rate. The CPU resource consumption cost of incremental storage is the product of the time required to complete the detection, calculation, merging and other operations of incremental difference data and the average CPU usage rate, as shown in formula (9), where T d , T p , T a They are the time required to complete the difference detection, difference calculation and difference merging operations respectively; C u =T c ×U u (8) C u =(T d +T p +T a )×U u (9) (25) Calculate the network resource consumption cost of the direct storage strategy and the incremental storage strategy for this version of data, that is, the network bandwidth resources consumed by storing this version of data, using C n It means that the network resource consumption cost of the direct storage strategy is the quotient of the complete data volume transmitted during the storage process and the available network bandwidth, as shown in formula (10), where D f is the amount of data stored and transmitted, B n is the available network bandwidth. The network resource consumption cost of the incremental storage strategy is the quotient of the differential data volume transmitted during the storage process and the available network bandwidth, as shown in formula (11), where D d It is the difference in the amount of data stored and transmitted, which is usually smaller than the entire amount of data; (26) Calculate the total cost of the direct storage strategy and the incremental storage strategy for the version data, denoted by C. The total cost of the two storage strategies is calculated as the weighted sum of storage space cost, retrieval time cost, CPU resource consumption cost, and network resource consumption cost. As shown in formula (12), the sum of the weight factors of the four costs is 1. α, β, and γ are the weight factors of storage space cost, retrieval time cost, and CPU resource consumption cost, respectively. In the data version storage decision, storage cost and retrieval cost are mainly used. The proportion of each cost in the total cost can be adjusted according to the actual situation. The calculated total cost of direct storage is denoted by C. d The total incremental storage cost is represented by C i express; C=αC s +βC f +γC u +(1-α-β-γ)C n (12) (27) Calculate the version update frequency, that is, the frequency at which each version data is updated, using F uv It is calculated by the quotient of the number of times the data is updated in a certain time interval and the time difference, as shown in formula (13), where n v is the number of times version v is updated in the time interval [t1, t2]. A version update frequency threshold is set according to the actual situation, and F is used to represent the frequency of version update. ut express: (28) Calculate the version access frequency, that is, the frequency with which each version data is accessed, using F av It is calculated by the quotient of the number of times the data is accessed in a certain time interval and the time difference, as shown in formula (14), where a v is the number of times version v is accessed in the time interval [t1, t2]. A version access frequency threshold is set according to the actual situation, and F is used to represent the access frequency of the version v. at express: (29) Comprehensively consider the total cost, version update frequency, version access frequency and other factors of the direct storage strategy and the incremental storage strategy, make a decision on the data version storage strategy, and select the optimal storage method. Specifically, if the total cost of direct storage is greater than the total cost of incremental storage, the data update frequency is less than its threshold, and the data access frequency is also less than its threshold, that is, C d >C i 、F uv <F ut And F av <F at , then the incremental storage strategy is selected. On the contrary, if the total cost of direct storage is less than or equal to the total cost of incremental storage, the data update frequency is greater than or equal to its threshold, or the data access frequency is greater than or equal to its threshold, that is, C d ≤C i 、F av ≥F at or F av ≥F at , then select the direct storage strategy; (210) The result of executing the multi-version data storage strategy is that if the data version selects the direct storage strategy, the complete data version is directly stored in the database without performing difference or incremental processing. In the graph database Neo4j, a child node is created on the old version data node to store the complete data of the new version. If the data version selects the incremental storage strategy, only the differences or changes between adjacent versions are stored. In the graph database Neo4j, an incremental attribute is added to the old version data node to store the differences between adjacent versions.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the multi-version unstructured data storage method for machine learning applications is implemented as described in any one of claims 1 to 4 above.

6. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instruction is executed by the processor, the multi-version unstructured data storage method for machine learning applications as described in any one of claims 1 to 4 is implemented.

Images