Traceable video optimal storage data system based on AI

Abstract

The invention, which relates to the technical field of artificial intelligence and data storage management, discloses an AI-based traceable video optimal storage data system comprising a video uploading module, a video processing module, a video optimal storage module, a data tracing retrieval end and a user interface end. The video uploading module provides an entrance for a user or equipment to upload video data, and supports the uploading of high-definition videos of various formats and sources; it is ensured that video data are not lost in the uploading process, and breakpoint resume and batch uploading are supported; and carrying out preliminary verification on the uploaded video. Through the AI intelligent identification and classification technology, high-value data are stored preferentially, redundant data are compressed or deleted, the storage space occupied by invalid data is reduced, and the overall storage cost is remarkably reduced; by embedding the AI-driven content label and the metadata index, the data tracing process is more accurate and faster, so that the searching and calling efficiency of specific data is greatly improved, and the requirement of a user for efficient data positioning is met.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and data storage management, and in particular to an AI-based traceable video storage data system. Background Art

[0002] At present, with the rapid development of information technology, the medical industry's demand for large-scale multimedia data such as videos and images is increasing year by year. How to efficiently and economically store and manage massive data has become a problem that needs to be solved urgently.

[0003] Traditional data storage methods mainly rely on disk arrays and cloud storage. Although they can meet basic storage needs, they have the following significant defects when the amount of data continues to expand. In multimedia data storage, due to the large amount of data and the long storage cycle, traditional storage methods require a lot of disk space, resulting in high costs for storage equipment and maintenance. Data traceability usually relies on file indexes or timestamps and lacks an intelligent content indexing mechanism, resulting in low retrieval efficiency and difficulty in locating specific content in a timely manner. Therefore, there is room for improvement. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of the prior art and propose an AI-based traceable video storage data system. Its advantage is that it reduces the storage space occupied by invalid data and significantly reduces the overall storage cost.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The AI-based traceable video storage data system includes a video upload module, a video processing module, a video storage module, a data traceability retrieval terminal and a user interface terminal; the video upload module provides an entry for users or devices to upload video data, and supports uploading high-definition videos in multiple formats and sources; ensures that video data is not lost during the upload process, supports breakpoint continuation and batch uploading; performs preliminary verification of uploaded videos to check file integrity and format correctness;

[0007] The video processing module mainly analyzes the uploaded video, including frame extraction, format conversion and resolution adjustment; the video storage module optimizes the storage strategy according to the content characteristics and access frequency of the video, and stores important or popular videos on high-speed storage media first; uses AI algorithms to compress video data and delete duplicate data to save storage space; realizes the separation of hot and cold data, formulates data migration strategies, and optimizes the utilization of storage resources;

[0008] The data traceability and retrieval end includes full life cycle traceability, records the entire process information of video data from uploading, processing, storage to access, supports traceability query; provides efficient video retrieval function, supports search based on content, features and metadata; records all operation logs;

[0009] The user interface provides an intuitive interface for users to upload, browse, retrieve and manage video data; and integrates real-time video playback function. The user interface module also provides manual adjustment function and plays back videos of specific time periods as needed; it displays statistical information, usage and system performance indicators of video data.

[0010] The present invention is further configured such that the video processing module uses a video processing algorithm to perform calculations, and the calculation process includes receiving and preliminary verification of video data, video preprocessing, and video content analysis.

[0011] The present invention is further configured such that the video content analysis includes intelligent desensitization and key frame summarization, wherein the intelligent desensitization extracts picture frames containing text information and facial information in the video through text detection and face detection, and removes sensitive frames through the ffmpeg tool.

[0012] The present invention is further configured such that the key frame summary extracts representative key frames from the video to reduce the amount of data, and an unsupervised learning method is adopted to summarize the video using a deep summary network (DSN). The whole process is a continuous decision-making process, and the DSN is an encoding-decoding structure, wherein the encoder is a convolutional neural network capable of extracting video frame features, and the decoder is a bidirectional LSTM network capable of generating probabilities based on selected actions.

[0013] The present invention is further configured as follows: the DSN encoder is a convolutional neural network, which extracts visual features t=1T from a video frame t=1T of input length T; the decoder is a bidirectional recurrent neural network (BiRNN), the top of which is a fully connected layer; after the extracted t=1T is input into the decoder, the corresponding hidden state t=1T is generated; each ht is the connection of the previous hidden state htf and the next hidden state htb; in practice, GoogLeNet is used as a CNN model, and LSTM training is used to improve the performance of RNN.

[0014] The present invention is further configured such that when the LSTM is being trained, the DSN receives a reward R(S) to evaluate the generated summary; and the goal of the DSN is to continuously generate high-quality video summaries to maximize the reward; and the reward method includes a diversity reward Rdiv and a representative reward Rrep.

[0015] The present invention is further configured such that the formula for the diversity reward Rdiv is: y represents the selected frame, and d(xt,xt') is the diversification formula.

[0016] The present invention is further configured such that the expression of d(xt, xt') is:

[0017]

[0018] The present invention is further configured such that the representative reward Rrep formula is: Under this reward, the agent can select the frame closest to the cluster center of the feature space. Finally, Rdiv and Rrep work together to guide DSN learning: R(S) = Rdiv + Rrep.

[0019] The present invention is further configured such that the data tracing and retrieval end adopts a data tracing and retrieval algorithm, which includes metadata management and indexing technology, data sharding and distributed retrieval, vector search and similarity retrieval, caching and acceleration strategy, and visual tracing and interactive retrieval.

[0020] The beneficial effects of the present invention are:

[0021] 1. Significantly reduce storage costs: Through AI intelligent identification and classification technology, high-value data is stored first and redundant data is compressed or deleted, which reduces the storage space occupied by invalid data and significantly reduces the overall storage cost.

[0022] 2. Improve data retrieval efficiency: By embedding AI-driven content tags and metadata indexes, the data tracing process is more accurate and faster, which greatly improves the efficiency of finding and calling specific data, meeting users' needs for efficient data positioning.

[0023] 3. Realize high-precision data management: The invention realizes dynamic tracking and management of data. All storage, modification, access and other operations are traceable, which greatly improves the transparency of data management and facilitates the coordination of storage, management, retrieval and other functions.

[0024] 4. Enhanced data security and controllability: Through layered optimal storage management, important data is stored in a storage layer with a higher security level, ensuring the security of key data. At the same time, through a complete traceability chain, it facilitates the full monitoring and management of data use and modification.

[0025] 5. Strong applicability and easy expansion: The present invention has modular and expandable characteristics, can adapt to a variety of data storage needs and application scenarios, is easy to integrate into existing data management systems, has strong compatibility, and provides users with flexible storage management solutions.

[0026] In summary, the present invention not only optimizes the utilization rate of storage resources, but also significantly improves data traceability, management transparency, security, etc. compared with the existing technology, helping to realize intelligent and efficient modern data storage management. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a schematic diagram of the overall structure of the AI-based traceable video storage data system proposed by the present invention;

[0028] Figure 2 This is a schematic diagram of the video processing module of the AI-based traceable video storage data system proposed by the present invention;

[0029] Figure 3 This is a logical diagram of frame retrieval of the AI-based traceable video storage data system proposed by the present invention;

[0030] Figure 4 Schematic diagram of the BiRNN model architecture of the AI-based traceable video storage data system proposed in the present invention. DETAILED DESCRIPTION

[0031] The technical solution of this patent is further described in detail below in conjunction with specific implementation methods.

[0032] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0033] Reference Figure 1-3 , an AI-based traceable video storage data system, including a video upload module, a video processing module, a video storage module, a data traceability retrieval terminal and a user interface terminal; the video upload module provides an entry for users or devices to upload video data, and supports uploading high-definition videos in multiple formats and sources; ensures that video data is not lost during the upload process, supports breakpoint resumption and batch upload; performs preliminary verification of uploaded videos, and checks file integrity and format correctness;

[0034] The video processing module mainly analyzes the uploaded video, including frame extraction, format conversion and resolution adjustment; the video storage module optimizes the storage strategy according to the content characteristics and access frequency of the video, and stores important or popular videos on high-speed storage media first; uses AI algorithms to compress video data and delete duplicate data to save storage space; realizes the separation of hot and cold data, formulates data migration strategies, and optimizes the utilization of storage resources;

[0035] The data traceability and retrieval end includes full life cycle traceability, records the entire process information of video data from uploading, processing, storage to access, supports traceability query; provides efficient video retrieval function, supports search based on content, features and metadata; records all operation logs;

[0036] The user interface provides an intuitive interface for users to upload, browse, retrieve and manage video data; and integrates real-time video playback function. The user interface module also provides manual adjustment function and plays back videos of specific time periods as needed; it displays statistical information, usage and system performance indicators of video data.

[0037] In this embodiment, the video processing module uses a video processing algorithm to perform calculations, and the calculation process includes receiving and preliminary verification of video data, video preprocessing, and video content analysis.

[0038] Receiving and preliminary verification of video data

[0039] Data integrity check: Verify whether the video file is damaged or incomplete during the upload process to ensure the integrity of the file.

[0040] Format and encoding verification: Check the format and encoding method of the video file to ensure that it complies with the standards supported by the system, such as MP4, AVI, MKV, etc.

[0041] Video Preprocessing

[0042] Format conversion: Convert videos into a unified encoding format and container format to facilitate subsequent processing and storage.

[0043] Resolution adjustment: Adjust the video resolution as needed, such as generating versions with multiple resolutions (1080p, 720p, 480p) to suit different terminals and network conditions.

[0044] Frame rate normalization: Adjust the frame rate of the video and process the frames uniformly to facilitate consistency in analysis and playback.

[0045] Reference Figure 4 The video content analysis includes intelligent desensitization and key frame summary. The intelligent desensitization extracts the picture frames containing text information and facial information in the video through text detection and face detection, and removes the sensitive frames through the ffmpeg tool; the key frame summary extracts representative key frames from the video to reduce the amount of data, and uses an unsupervised learning method to summarize the video with a deep summary network (DSN). The whole process is a continuous decision-making process. DSN is an encoding-decoding structure, in which the encoder is a convolutional neural network that can extract video frame features, and the decoder is a bidirectional LSTM network that can generate probabilities based on selected actions.

[0046] The DSN encoder is a convolutional neural network, which extracts visual features t=1T from the input video frame t=1T of length T; the decoder is a bidirectional recurrent neural network (BiRNN), with a fully connected layer at the top; after the extracted t=1T is input to the decoder, the corresponding hidden state t=1T is generated; each ht is the connection of the previous hidden state htf and the next hidden state htb; in practice, GoogLeNet is used as the CNN model, and LSTM training is used to improve the performance of RNN; when the LSTM is trained, DSN will receive a reward R(S) to evaluate the generated summary; and the goal of DSN is to continuously generate high-quality video summaries to maximize the reward; the reward method includes diversity reward Rdiv and representative reward Rrep; the formula of the diversity reward Rdiv is: y represents the selected frame, d(xt,xt') is a diversified formula; the expression of d(xt,xt') is: The representative reward Rrep formula is: Under this reward, the agent can select the frame closest to the cluster center of the feature space. Finally, Rdiv and Rrep work together to guide DSN learning: R(S) = Rdiv + Rrep.

[0047] It is worth mentioning that the data traceability retrieval end adopts a data traceability retrieval algorithm, which includes metadata management and indexing technology, data sharding and distributed retrieval, vector search and similarity retrieval, caching and acceleration strategy, and visual traceability and interactive retrieval.

[0048] Metadata management and indexing technology

[0049] Metadata generation and storage: Generate and store metadata for video data, including basic video attributes (time, location, format), content features (labels, objects), and operation records (upload time, processing steps, access history, etc.).

[0050] Hierarchical metadata: Metadata can be divided into different levels such as time, space, and content, making it easier for the system to trace data in different dimensions.

[0051] Inverted index: Applicable to full-text search and tag search. Inverted index associates keywords or tags with video data to quickly locate files containing specific keywords.

[0052] Vector index: Based on the feature vector of the video, use the vector index (FAISS library) to quickly search for similarity. Used to retrieve videos or images with similar content

[0053] Data Sharding and Distributed Retrieval

[0054] Horizontal sharding: shard large-scale data according to a certain attribute (time or category) so that the data is distributed across multiple nodes or storage areas for easy parallel retrieval.

[0055] Vertical sharding: Sharding by data attributes, video metadata, content features, and operation records are stored separately. This allows only relevant attributes to be queried during retrieval, improving efficiency.

[0056] Distributed file system: HDFS (Hadoop Distributed File System) is used to store large amounts of video files for easy parallel access.

[0057] Distributed index system: Use Elasticsearch to build distributed indexes to achieve real-time search and traceability of large-scale data.

[0058] Vector Search and Similarity Retrieval

[0059] Feature vector generation: Use the deep learning model (DSN) to extract features from video content and convert the video content into a high-dimensional feature vector.

[0060] Vector library construction: Use a specialized vector retrieval library (FAISS) to build a vector index library.

[0061] Similarity measurement: Retrieve the data most similar to the target video based on indicators such as Euclidean distance and cosine similarity. Used to trace video clips with similar content.

[0062] Caching and acceleration strategies

[0063] Multi-level cache: Use multi-level cache (memory cache, SSD cache) to save metadata and indexes of hot videos, reduce disk I / O, and improve response speed.

[0064] Cache elimination strategy: Combine LRU (least recently used) and LFU (least frequently used) strategies to dynamically update cache data and maintain cache validity.

[0065] Visual traceability and interactive retrieval

[0066] Traceability path diagram: displays the operation path and flow of video data, including uploading, processing, storage and other information.

[0067] Timeline retrieval: A timeline-based interface presentation that supports users to query video operation records by timeline.

[0068] Data tracking dashboard: displays the real-time status and access history of video data in the system, making it easier for administrators to monitor.

[0069] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. AI-based traceable video storage data system, characterized by: It includes a video upload module, a video processing module, a video storage module, a data tracing and retrieval terminal and a user interface terminal; the video upload module provides an entry for users or devices to upload video data, and supports uploading high-definition videos in multiple formats and sources; Ensure that video data is not lost during the upload process, support breakpoint resuming and batch uploading; perform preliminary verification on uploaded videos to check file integrity and format correctness; The video processing module mainly analyzes the uploaded video, including frame extraction, format conversion and resolution adjustment; the video storage module optimizes the storage strategy according to the content characteristics and access frequency of the video, and stores important or popular videos on high-speed storage media first; uses AI algorithms to compress and delete duplicate data of video data to save storage space; Separate hot and cold data, formulate data migration strategies, and optimize storage resource utilization; The data traceability retrieval terminal includes full life cycle traceability, records the entire process information of video data from uploading, processing, storage to access, and supports traceability query; Provides efficient video retrieval function, supports searches based on content, features, and metadata; records all operation logs; The user interface provides an intuitive interface for users to upload, browse, retrieve and manage video data; and integrates real-time video playback function. The user interface module also provides manual adjustment function and plays back videos of specific time periods as needed; it displays statistical information, usage and system performance indicators of video data.

2. The AI-based traceable video storage data system according to claim 1 is characterized in that: The video processing module uses a video processing algorithm to perform calculations, and the calculation process includes receiving and preliminary verification of video data, video preprocessing, and video content analysis.

3. The AI-based traceable video storage data system according to claim 2 is characterized in that: The video content analysis includes intelligent desensitization and key frame summary. The intelligent desensitization extracts the picture frames containing text information and facial information in the video through text detection and face detection, and removes sensitive frames through the ffmpeg tool.

4. The AI-based traceable video storage data system according to claim 3 is characterized in that: The key frame summary extracts representative key frames from the video to reduce the amount of data, adopts an unsupervised learning method, and summarizes the video with a deep summary network (DSN). The whole process is a continuous decision-making process. The DSN is an encoding-decoding structure, in which the encoder is a convolutional neural network that can extract video frame features, and the decoder is a bidirectional LSTM network that can generate probabilities based on selected actions.

5. The AI-based traceable video storage data system according to claim 4 is characterized in that: The DSN encoder is a convolutional neural network that extracts visual features t=1T from an input video frame t=1T of length T; The decoder is a bidirectional recurrent neural network (BiRNN) with a fully connected layer on top. After the extracted t=1T is input into the decoder, the corresponding hidden state t=1T is generated. Each ht is the connection of the previous hidden state htf and the next hidden state htb. In practice, GoogLeNet is used as the CNN model, and LSTM training is used to improve the performance of RNN.

6. The AI-based traceable video storage data system according to claim 5 is characterized in that: When the LSTM is being trained, the DSN receives a reward R(S) to evaluate the generated summary. The goal of the DSN is to continuously generate high-quality video summaries to maximize the reward. The reward method includes a diversity reward Rdiv and a representative reward Rrep.

7. The AI-based traceable video storage data system according to claim 6 is characterized in that: The formula for the diversity bonus Rdiv is: y represents the selected frame, and d(xt,xt') is the diversification formula.

8. The AI-based traceable video storage data system according to claim 7 is characterized in that: The expression of d(xt,xt') is:

9. The AI-based traceable video storage data system according to claim 8 is characterized in that: The representative reward Rrep formula is: Under this reward, the agent can select the frame closest to the cluster center of the feature space. Finally, Rdiv and Rrep work together to guide DSN learning: R(S) = Rdiv + Rrep.

10. The AI-based traceable video storage data system according to any one of claims 1 to 9, characterized in that: The data tracing and retrieval end adopts a data tracing and retrieval algorithm, which includes metadata management and indexing technology, data sharding and distributed retrieval, vector search and similarity retrieval, cache and acceleration strategy, and visual tracing and interactive retrieval.

Images