Data optimization storage method based on artificial intelligence and big data system

Abstract

The invention discloses a data optimization storage method based on artificial intelligence and a big data system, and relates to the technical field of data storage, and the method comprises the steps: obtaining files, classifying the files into pictures, documents and videos, and extracting the features of the files through different methods; through a multi-modal learning technology, mining semantic association among the pictures, the documents and the videos, and constructing an association graph; performing compression, hierarchical storage and index optimization on the data according to the semantic association and the association graph; according to the method, the data is stored, the storage state is monitored in real time, the storage performance indexes are collected, the model is optimized according to the performance indexes, fusion storage of pictures, documents and videos is achieved, and subsequent file query is facilitated by intelligently analyzing data features and mining semantic association among the data.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to data optimization storage methods and big data systems based on artificial intelligence. Background Technology

[0002] With the rapid development of information technology, the amount of data is growing explosively, especially multimedia data such as pictures, documents, and videos. Traditional storage methods usually store different types of data separately, resulting in low storage efficiency, difficult retrieval, and high costs. In addition, there are rich semantic relationships between different types of data, but traditional methods cannot make full use of these relationships to optimize storage. Therefore, there is an urgent need for storage methods that can integrate different types of data to improve storage efficiency, reduce costs, and enhance data accessibility and manageability. To this end, we propose an intelligent service robot management and control system based on data analysis. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides an intelligent service robot management system based on data analysis, including: S1, obtaining files, classifying files into images, documents, and videos, and extracting features from the files using different methods;

[0004] S2. Through multimodal learning techniques, extract semantic relationships between images, documents, and videos, and construct a relationship graph;

[0005] S3. Based on semantic association and association graph, compress the data, store it in layers, and optimize the index;

[0006] S4. Store the data, monitor the storage status in real time, collect storage performance indicators, and optimize the model based on the performance indicators.

[0007] Furthermore, the process of obtaining files and classifying them into images, documents, and videos, and extracting features from the files using different methods, S1 includes: identifying the obtained files, classifying them into text documents, images, and videos, and performing data cleaning on the files to remove abnormal data;

[0008] Convolutional neural networks are used to extract visual features from images, natural language processing techniques are used to extract semantic features from documents, and a Transformer-based model is used to extract spatiotemporal features from videos.

[0009] Furthermore, the step of mining semantic associations between images, documents, and videos and constructing association graph S2 through multimodal learning technology includes: using a contrastive learning method to map the features of images, documents, and videos to the same embedding space, and constructing positive sample pairs and negative sample pairs;

[0010] By minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs, the model parameters are optimized so that samples of different modalities can be mapped to a semantically consistent embedding space.

[0011] The preprocessed and feature-extracted image, document, and video data are input into the contrastive learning model. The model continuously adjusts its parameters, learns how to map features of different modalities to the same embedding space, and mines the semantic relationships between them. Through multiple iterations of training, the model converges, and the optimized embedding space and model parameters are obtained.

[0012] Images, documents, and videos are treated as nodes in a graph, and the feature vector of each node is a feature vector mapped to the same embedding space after being processed by a contrastive learning model.

[0013] Cosine similarity is used to calculate the similarity between the feature vectors of two nodes;

[0014] Set a threshold. Based on the calculated semantic association strength, when the semantic association strength is greater than the threshold, add an edge between the two nodes. The weight of the edge is the semantic association strength.

[0015] Use graph data structures to store node and edge information, optimize the graph, and use graph visualization tools to visualize the constructed relational graph.

[0016] Furthermore, the step of compressing, storing hierarchically, and optimizing the index of data based on semantic association and association graph S3 includes: selecting the optimal compression algorithm based on data characteristics and association strength. The method for selecting the compression algorithm is as follows: if the association strength between data is high, a prediction-based compression algorithm is used; if the document content and image content are highly correlated, the text features of the document and the visual features of the image are jointly encoded; if the association strength between data is low, each data type is compressed independently.

[0017] Based on the access frequency and importance of the data, data is allocated to hot storage, warm storage, or cold storage media. Hot storage offers high read / write speeds and high costs; warm storage offers medium read / write speeds and medium costs; and cold storage offers low read / write speeds and low costs. The allocation method is as follows:

[0018] Define access frequency weight W f And data importance weight W i ,

[0019] For each data object, a score is assigned based on its access frequency and importance. The access frequency is categorized as high-frequency, medium-frequency, and low-frequency, and the importance is categorized as high, medium, and low. The comprehensive score is calculated using the following formula:

[0020] ;

[0021] Where S is the overall score, the S f S represents the frequency of access. i For data importance scores;

[0022] Based on the association graph, a multimodal index is generated to optimize data retrieval efficiency.

[0023] Furthermore, the multimodal index construction step is as follows: obtaining the feature vector of each node from the association graph;

[0024] Calculate the similarity of each node to other nodes to obtain the edge weights;

[0025] The feature vector and list of similar nodes for each node are stored in the index, and the nodes and edges are stored in the database, with the edge weights stored as attributes.

[0026] The edge weight We is calculated using cosine similarity:

[0027] ;

[0028] Where A and B are the feature vectors of two nodes, and ||A|| and ||B|| are the norms of the vectors;

[0029] The method for optimizing data retrieval efficiency is as follows: cluster nodes using a clustering algorithm to reduce the retrieval range; quickly locate relevant nodes through the association graph based on the query semantics; and sort them using edge weights.

[0030] Furthermore, the process of storing data, monitoring storage status in real time, collecting storage performance metrics, and optimizing model S4 based on performance metrics includes: compressing data, storing it in layers, and creating indexes according to the generated strategy.

[0031] Monitor storage status in real time and collect performance metrics;

[0032] Establish an I / O workload analysis model to identify performance bottlenecks by analyzing the type, pattern, and characteristics of the workload;

[0033] The collected performance metrics data are used as input to the optimization model. The model learns and adjusts based on this data, and optimizes the storage system based on the model's output.

[0034] Evaluate whether the performance of the optimized storage system has improved, continuously monitor the performance of the storage system to ensure that the optimization effect is maintained, and further adjust the model based on new data feedback.

[0035] AI-based data optimization and storage big data system, including,

[0036] The feature extraction module identifies the format of the acquired files, preprocesses the files, and then extracts features from the preprocessed files.

[0037] Establish an association module to create associations between various files based on the extracted features, and set up an association graph;

[0038] The storage module compresses data, stores it in layers according to its importance, and stores data in association.

[0039] The monitoring module monitors data storage, collects intelligent storage performance indicators, and optimizes storage based on these indicators.

[0040] A computer device includes a memory and a processor, the memory storing a computer program, wherein: when the computer program is executed by the processor, it implements any step of the data optimization storage method based on artificial intelligence as described in the first aspect of the present invention.

[0041] A computer-readable storage medium having a computer program stored thereon, wherein, when executed by a processor, the computer program implements any step of the data optimization storage method based on artificial intelligence as described in the first aspect of the present invention.

[0042] The beneficial effects of this invention are reflected in:

[0043] 1. This invention achieves integrated storage of images, documents, and videos. By intelligently analyzing data features, mining semantic relationships between data, and dynamically optimizing storage strategies, it enables efficient integrated storage of different types of data. Furthermore, it rationally allocates storage resources based on data value and access needs, using high-cost, high-performance storage for critical data and low-cost storage for less important data, thereby improving the overall performance of the storage system. Data compression is achieved by re-encoding data using specific algorithms, reducing data storage space and significantly saving costs. Attached Figure Description

[0044] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0045] Figure 1 This is a flowchart illustrating the data optimization and storage method based on artificial intelligence in Example 1. Detailed Implementation

[0046] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0047] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0048] like Figure 1 As shown, the intelligent service robot management system based on data analysis includes the following steps:

[0049] S1. Obtain the files and classify them into images, documents, and videos. Extract the features of the files using different methods.

[0050] S2. Through multimodal learning techniques, extract semantic relationships between images, documents, and videos, and construct a relationship graph;

[0051] S3. Based on semantic association and association graph, compress the data, store it in layers, and optimize the index;

[0052] S4. Store the data, monitor the storage status in real time, collect storage performance indicators, and optimize the model based on the performance indicators.

[0053] In this embodiment, obtaining a file and classifying it into images, documents, and videos, and extracting features from the file using different methods, S1 includes: identifying the obtained file, classifying the file into text documents, images, and videos, and cleaning the file data to remove abnormal data;

[0054] Convolutional neural networks are used to extract visual features from images, natural language processing techniques are used to extract semantic features from documents, and a Transformer-based model is used to extract spatiotemporal features from videos.

[0055] In this embodiment, the semantic association between images, documents and videos is mined and the association graph S2 is constructed by using multimodal learning technology. This includes: using a contrastive learning method to map the features of images, documents and videos to the same embedding space and constructing positive sample pairs and negative sample pairs.

[0056] By minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs, the model parameters are optimized so that samples of different modalities can be mapped to a semantically consistent embedding space.

[0057] The preprocessed and feature-extracted image, document, and video data are input into the contrastive learning model. The model continuously adjusts its parameters, learns how to map features of different modalities to the same embedding space, and mines the semantic relationships between them. Through multiple iterations of training, the model converges, and the optimized embedding space and model parameters are obtained.

[0058] Images, documents, and videos are treated as nodes in a graph, and the feature vector of each node is a feature vector mapped to the same embedding space after being processed by a contrastive learning model.

[0059] Cosine similarity is used to calculate the similarity between the feature vectors of two nodes;

[0060] Set a threshold. Based on the calculated semantic association strength, when the semantic association strength is greater than the threshold, add an edge between the two nodes. The weight of the edge is the semantic association strength.

[0061] Use graph data structures to store node and edge information, optimize the graph, and use graph visualization tools to visualize the constructed relational graph.

[0062] In this embodiment, the data compression, hierarchical storage, and index optimization based on semantic association and association graph S3 include: selecting the optimal compression algorithm based on data characteristics and association strength. The method for selecting the compression algorithm is as follows: if the association strength between data is high, a prediction-based compression algorithm is used; if the document content and image content are highly correlated, the text features of the document and the visual features of the image are jointly encoded; if the association strength between data is low, each data type is compressed independently.

[0063] Based on the frequency and importance of data access, data is allocated to hot storage, warm storage, or cold storage media. Hot storage offers high read / write speeds but incurs high costs; warm storage offers medium read / write speeds and incurs medium costs; and cold storage offers low read / write speeds but incurs low costs. The allocation method is as follows:

[0064] Define access frequency weight W f And data importance weight W i ,

[0065] For each data object, a score is assigned based on its access frequency and importance. Access frequency is categorized as high-frequency, medium-frequency, and low-frequency, and importance is categorized as high, medium, and low. The overall score is calculated using the following formula:

[0066] ;

[0067] Where S is the overall score, S f S represents the frequency of access. i For data importance scores;

[0068] Based on the association graph, a multimodal index is generated to optimize data retrieval efficiency.

[0069] In this embodiment, the multimodal index construction steps are as follows: obtain the feature vector of each node from the association graph;

[0070] Calculate the similarity of each node to other nodes to obtain the edge weights;

[0071] The feature vector and list of similar nodes for each node are stored in the index, and the nodes and edges are stored in the database, with the edge weights stored as attributes.

[0072] The edge weight We is calculated using cosine similarity:

[0073] ;

[0074] Where A and B are the feature vectors of two nodes, and ||A|| and ||B|| are the norms of the vectors;

[0075] The method to optimize data retrieval efficiency is as follows: use clustering algorithms to cluster nodes to reduce the retrieval scope, quickly locate relevant nodes through the association graph based on the query semantics, and sort them using edge weights.

[0076] Furthermore, the data is stored, and the storage status is monitored in real time. Storage performance metrics are collected, and model S4 is optimized based on the performance metrics, including: compressing the data, tiering storage, and creating indexes according to the generated strategy.

[0077] Monitor storage status in real time and collect performance metrics;

[0078] Establish an I / O workload analysis model to identify performance bottlenecks by analyzing the type, pattern, and characteristics of the workload;

[0079] The collected performance metrics data are used as input to the optimization model. The model learns and adjusts based on this data, and optimizes the storage system based on the model's output.

[0080] Evaluate whether the performance of the optimized storage system has improved, continuously monitor the performance of the storage system to ensure that the optimization effect is maintained, and further adjust the model based on new data feedback.

[0081] AI-based data optimization and storage big data system, including,

[0082] The feature extraction module identifies the format of the acquired files, preprocesses the files, and then extracts features from the preprocessed files.

[0083] Establish an association module to create associations between various files based on the extracted features, and set up an association graph;

[0084] The storage module compresses data, stores it in layers according to its importance, and stores data in association.

[0085] The monitoring module monitors data storage, collects intelligent storage performance indicators, and optimizes storage based on these indicators.

[0086] A computer device suitable for an AI-based data optimization storage method includes: a memory and a processor; the memory stores computer-executable instructions, and the processor executes the computer-executable instructions to implement the AI-based data optimization storage method as proposed in the above embodiments.

[0087] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0088] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the data optimization storage method based on artificial intelligence proposed in the above embodiment. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0089] In summary, this invention achieves efficient integrated storage of different types of data by intelligently analyzing data features, mining semantic relationships between data, and dynamically optimizing storage strategies. Furthermore, it rationally allocates storage resources based on data value and access needs, using high-cost, high-performance storage for critical data and low-cost storage for less important data, thereby improving the overall performance of the storage system. Data compression is achieved by re-encoding data using specific algorithms, reducing data storage space and significantly saving costs.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A data optimization and storage method based on artificial intelligence, characterized in that, include: S1. Obtain the files and classify them into images, documents, and videos. Extract the features of the files using different methods. S2. Through multimodal learning techniques, extract semantic relationships between images, documents, and videos, and construct a relationship graph; S3. Based on semantic association and association graph, compress the data, store it in layers, and optimize the index; S4. Store the data, monitor the storage status in real time, collect storage performance indicators, and optimize the model based on the performance indicators.

2. The data optimization and storage method based on artificial intelligence as described in claim 1, characterized in that: The process of obtaining files and classifying them into images, documents, and videos, and extracting features from the files using different methods, S1 includes: identifying the obtained files, classifying them into text documents, images, and videos, and cleaning the files to remove abnormal data. Convolutional neural networks are used to extract visual features from images, natural language processing techniques are used to extract semantic features from documents, and a Transformer-based model is used to extract spatiotemporal features from videos.

3. The data optimization and storage method based on artificial intelligence as described in claim 2, characterized in that: The step of mining semantic associations between images, documents, and videos and constructing association graphs S2 through multimodal learning technology includes: using a contrastive learning method to map the features of images, documents, and videos to the same embedding space and constructing positive sample pairs and negative sample pairs; By minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs, the model parameters are optimized so that samples of different modalities can be mapped to a semantically consistent embedding space. The preprocessed and feature-extracted image, document, and video data are input into the contrastive learning model. The model continuously adjusts its parameters, learns how to map features of different modalities to the same embedding space, and mines the semantic relationships between them. Through multiple iterations of training, the model converges, and the optimized embedding space and model parameters are obtained. Images, documents, and videos are treated as nodes in a graph, and the feature vector of each node is a feature vector mapped to the same embedding space after being processed by a contrastive learning model. Cosine similarity is used to calculate the similarity between the feature vectors of two nodes; Set a threshold. Based on the calculated semantic association strength, when the semantic association strength is greater than the threshold, add an edge between the two nodes. The weight of the edge is the semantic association strength. Use graph data structures to store node and edge information, optimize the graph, and use graph visualization tools to visualize the constructed relational graph.

4. The data optimization and storage method based on artificial intelligence as described in claim 3, characterized in that: The step of compressing, storing in layers, and optimizing the index of data based on semantic association and association graph S3 includes: selecting the optimal compression algorithm based on data characteristics and association strength. The method for selecting the compression algorithm is as follows: if the association strength between data is high, a prediction-based compression algorithm is used; if the document content and image content are highly correlated, the text features of the document and the visual features of the image are jointly encoded; if the association strength between data is low, each data type is compressed independently. Based on the access frequency and importance of the data, data is allocated to hot storage, warm storage, or cold storage media. Hot storage offers high read / write speeds and high costs; warm storage offers medium read / write speeds and medium costs; and cold storage offers low read / write speeds and low costs. The allocation method is as follows: Define access frequency weight W f And data importance weight W i , For each data object, a score is assigned based on its access frequency and importance. The access frequency is categorized as high-frequency, medium-frequency, and low-frequency, and the importance is categorized as high, medium, and low. The comprehensive score is calculated using the following formula: ； Where S is the overall score, the S f S is the access frequency score. i For data importance scores; Based on the association graph, a multimodal index is generated to optimize data retrieval efficiency.

5. The data optimization and storage method based on artificial intelligence as described in claim 3, characterized in that: The multimodal index construction steps are as follows: obtain the feature vector of each node from the association graph; calculate the similarity of each node with other nodes to obtain the edge weights; The feature vector and list of similar nodes for each node are stored in the index, and the nodes and edges are stored in the database, with the edge weights stored as attributes. Edge weight W e Calculated using cosine similarity: ； Where A and B are the feature vectors of two nodes, and ||A|| and ||B|| are the norms of the vectors; The method for optimizing data retrieval efficiency is as follows: cluster nodes using a clustering algorithm to reduce the retrieval range; quickly locate relevant nodes through the association graph based on the query semantics; and sort them using edge weights.

6. The data optimization and storage method based on artificial intelligence as described in claim 4, characterized in that: The process of storing data, monitoring storage status in real time, collecting storage performance indicators, and optimizing model S4 based on performance indicators includes: compressing data, storing it in layers, and creating indexes according to the generated strategy. Monitor storage status in real time and collect performance metrics; Establish an I / O workload analysis model to identify performance bottlenecks by analyzing the type, pattern, and characteristics of the workload; The collected performance metrics data are used as input to the optimization model. The model learns and adjusts based on this data, and optimizes the storage system based on the model's output. Evaluate whether the performance of the optimized storage system has improved, continuously monitor the performance of the storage system to ensure that the optimization effect is maintained, and further adjust the model based on new data feedback.

7. A data optimization and storage big data system based on artificial intelligence, based on the data optimization and storage method based on artificial intelligence as described in any one of claims 1-6, characterized in that: include, The feature extraction module identifies the format of the acquired files, preprocesses the files, and then extracts features from the preprocessed files. Establish an association module to create associations between various files based on the extracted features, and set up an association graph; The storage module compresses data, stores it in layers according to its importance, and stores data in association. The monitoring module monitors data storage, collects storage performance metrics, and optimizes storage based on these metrics.

8. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the data optimization storage method based on artificial intelligence as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the data optimization storage method based on artificial intelligence as described in any one of claims 1-6.

Images