Archive data retrieval method, system and device

Abstract

The invention relates to the technical field of document data retrieval, and discloses an archive data retrieval method, system and device.Firstly, multiple pieces of archive data in a system are cleaned and standardized, document data are preprocessed, and document data processing specifically comprises text data, picture data and video data; semantic representation is carried out on the processed archive data, specifically, a deep learning model is used for creating semantic vector representation for a document; the expressions of the text, the image and the video are fused into a unified vector space, data of various data types of the text, the image and the video are integrated, and cross-modal retrieval is carried out; and outputting a retrieval result according to the similarity ranking. According to the method, the query intention of the user and the content of the document can be more accurately understood, keyword extraction and sentiment analysis can be carried out, the system can be helped to more comprehensively analyze the document, and therefore the correlation of retrieval results is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of archive data retrieval, and in particular to an archive data retrieval method, system and device. Background Art

[0002] The current archival data retrieval method and system have been gradually developed under the background of the rapid development of information technology, especially computer technology, aiming to improve the utilization efficiency and management level of archival information. The background technology of the current archival data retrieval method and system specifically includes keyword search:

[0003] Principle: Search the archive database for records containing the keywords entered by the user.

[0004] Features: fast and direct, suitable for search needs with clear keywords.

[0005] Application: Widely used in various file management software, such as searching based on keywords such as file title, content summary, etc.

[0006] Attribute filter:

[0007] Principle: Filter according to the attributes of the file (such as file type, number, status, department, etc.) to narrow the search scope.

[0008] Features: Accurate and efficient, suitable for queries with specific attribute requirements for archives.

[0009] Application: On the interface of the archive management software, users can select different attributes for combined screening to quickly locate the target archive.

[0010] Directory Navigation:

[0011] Principle: Use the hierarchical structure of the archive directory to locate the required archive by clicking on the directory items step by step.

[0012] Features: intuitive and easy to use, suitable for users who have a certain understanding of file structure.

[0013] Application: In file management software, a directory navigation function is usually provided to help users quickly find the location of the target file.

[0014] Tag Management:

[0015] Principle: Different labels are given to files, and users can search for files with the same label by selecting the label.

[0016] Features: Flexible and diverse, files can be labeled according to different classification standards (such as year, department, project, etc.).

[0017] Application: In the archive management software, users can customize tags, mark and classify archives for quick search and management.

[0018] Full text search:

[0019] Principle: Full-text indexing of archive contents, users can search for text content in archives by entering keywords.

[0020] Features: comprehensive and in-depth, able to cover the entire content of the archives and improve the accuracy and comprehensiveness of searches.

[0021] Application: Full-text search is particularly important when you need to find specific text content in an archive.

[0022] However, the retrieval algorithm is limited. Keyword matching is not accurate: relying on keyword matching may ignore documents with similar meanings but different wording. Insufficient synonym processing: the system may not be able to recognize the different meanings of synonyms or polysemous words. Limited context understanding ability: although supported by natural language processing technology, it is still difficult to handle complex query intent; and in the scope of multimodal retrieval, there are cross-modal consistency issues. When dealing with the association between text and non-text (such as images and videos), the system is difficult to ensure consistency.

[0023] Therefore, in order to solve the above problems, a method, system and device for archival data retrieval are needed. Summary of the invention

[0024] The purpose of the present invention is to provide an archival data retrieval method, system and device. The present invention can more accurately understand the user's query intention and the content of the document, and keyword extraction and sentiment analysis can help the system analyze the document more comprehensively, thereby improving the relevance of the retrieval results. And enhance the retrieval speed, distributed storage and cloud computing technology can significantly improve the speed of data processing, making large-scale archival retrieval faster, and efficient index structure and parallel processing capabilities help to speed up the retrieval process. Through the cross-modal fusion module, the system can process and retrieve data of various types including text, images, videos, etc. It allows users to query related data of other modalities through data of any modality.

[0025] The present invention is achieved in that:

[0026] The present invention provides an archive data retrieval method, which is specifically performed according to the following steps:

[0027] S1: First, multiple archive data in the system are cleaned and standardized, and the document data is preprocessed, wherein the document data processing specifically includes text, picture and video data;

[0028] Please follow the steps below:

[0029] S1.1: Perform data cleaning, remove noise, correct spelling errors, unify date and time formats, and denoise as shown in formula (1);

[0030]

[0031] Among them, y i is the i-th element in the new sequence after moving average processing; n is the window size of the moving average, the number of data points for calculating the average value;

[0032] x j is the jth element in the original data sequence;

[0033] j=i-(n-1) / 2 to i+(n-1) / 2 is the range of data points involved in the average calculation;

[0034] Specifically, the Levenshtein distance is used to calculate the difference between two strings to correct spelling errors, and the closest correct spelling is applied; as shown in formula (2);

[0035]

[0036] Where D represents the Levenshtein distance function and V is the vocabulary;

[0037] S1.2: All document data have the same metadata format; standardize the data, and specifically use a standardization method to remove, such as formula (3);

[0038]

[0039] Where x is the original data point, μ is the mean, and σ is the standard deviation;

[0040] S1.3: Normalize the data date and convert all dates into a unified format using ISO 8601, as shown in formula (4);

[0041] Normalized Date=ConvertToISO8601(Original Date) Formula (4)

[0042] S2: semantically represent the processed archival data, specifically using a deep learning model to create a semantic vector representation for the document;

[0043] To perform semantic representation, please follow the steps below:

[0044] S2.1: Text encoding uses the Transformer model, BERT or its variants to obtain the semantic representation of the text; specifically, as shown in formula (5);

[0045]

[0046] Among them, t i is a text sequence T = [t1, t2, ..., t n ], use the BERT model to obtain the embedding vector e of each word i ;

[0047] S2.2: Encode the image and use the convolutional neural network (CNN) to extract image features; use the pre-trained ResNet model to encode the image I and obtain the feature vector v I , specifically as formula (6);

[0048] v I =ResNet(I) Formula (6)

[0049] S2.3: Perform video encoding, use 3DCNN or LSTM to extract features of video frames, use the 3DCNN model to encode frames in the video V, and then average the feature vectors of all frames; as shown in formula (7);

[0050]

[0051] Among them, f j is the jth frame in the video, and 3DCNN(·) is the forward propagation function of the 3D CNN model.

[0052] S3: Fusion of text, image, and video representations into a unified vector space, and integration of data from multiple data types such as text, image, and video for cross-modal retrieval;

[0053] Please follow the steps below:

[0054] S3.1: Create a joint table by weighted averaging to fuse the representations of text, image, and video into a unified vector space; as shown in formula (8);

[0055] v joint =αv T +βv I +γv V Formula (8)

[0056] Among them, α, β, and γ are weight coefficients;

[0057] S3.2: The user query is encoded by the encoder in step S3.1; as shown in formula (9);

[0058] q=Encoder(Q) Formula (9)

[0059] Where Q is the text query data provided by the user; Encoder(·) is the encoder function;

[0060] S3.3: Use cosine similarity to measure the joint representation of query vector q and document v joint The similarity between them; as shown in formula (10);

[0061]

[0062] Among them, q·V joint is the product of vectors, ||q|| and ||v joint || are the Euclidean norms of the vectors respectively;

[0063] S3.4: Output the search results according to the similarity ranking.

[0064] S4: Output the search results based on similarity ranking.

[0065] Furthermore, in step S2 of the present invention, the text processing module also performs sentiment analysis on the text content of the document to identify the sentiment tendency of the document, which is specifically performed in the following steps:

[0066] S5.1: Use TF-IDF to convert the text into a numerical vector; and calculate the term frequency-inverse document frequency, as shown in equations (11)-(12);

[0067] IF-IDF(t,d,D)=TF(t,d)=IDF(t,D) Formula (11)

[0068] Among them, TF(t, d) is the frequency of word t in document d, and IDF(t, D) is the frequency of word t in the inverse document;

[0069]

[0070] Where N is the total number of documents, n t is the number of documents containing word t;

[0071] S5.2: Calculate the sentiment score of the document and perform sentiment classification, as shown in formula (13)-formula (14);

[0072] y=σ(w T x+b) Formula (13)

[0073] Among them, x is the feature quantity, w is the weight vector, b is the bias term, and σ(·) is the sigmoid function;

[0074]

[0075] Where n is the number of words in the document, w i For a given word ti The weight of TF-IDF(t i , d, D) is the word t i TF-IDF value in document d.

[0076] Furthermore, the present invention also includes the following method for cross-modal retrieval:

[0077] S6.1: First, construct an index structure for the processed data, where the index structure is a KD tree structure; the index structure is constructed based on recursive partitioning, each time a dimension is selected and the median is found on the dimension as the split point; the training data set is used as the basic data for constructing the index structure; and the document data identified in step S1 is used to maintain the index, and the real-time data stream continuously updates the index structure.

[0078] S6.2: When a document query is received, the index structure is used to quickly locate the possible candidate set; accurate distance calculations are performed on the elements in the candidate set to find the nearest neighbors. When a query is received, the tree is traversed recursively, and each time the coordinates of the query point are used to decide whether to enter the left subtree or the right subtree. If a leaf node is encountered, all data points in the node are recorded;

[0079] When backtracking, check whether the subtree on the other side may contain better candidate points;

[0080] Calculate the distance between the query point q and the data point p. Specifically, perform accurate distance calculation to find the true nearest neighbor from the candidate set. First, calculate the exact distance between each data point in the candidate set and the query point, and then find the data point with the minimum distance as the nearest neighbor as shown in formula (15);

[0081]

[0082] Among them, p i and q i are the values ​​of data point,p,and query point,q,on the i-th dimension, respectively, and k is the dimension of the data point;

[0083] Furthermore, the present invention provides an archive data retrieval system, including a data preprocessing module, wherein the data preprocessing module includes data cleaning, specifically text cleaning, removal of irrelevant characters and punctuation marks, spelling correction, and data standardization: unifying the format of the document, including date format, uppercase and lowercase;

[0084] Includes a text processing module that performs text encoding and obtains the semantic representation of documents through Transformer, BERT or RoBERTa;

[0085] It also performs keyword extraction, extracts keywords from documents through TF-IDF to enhance retrieval results; and performs sentiment analysis to identify the sentiment tendency of documents;

[0086] Image processing module, which encodes images in documents and extracts image features using a pre-trained CNN model;

[0087] And perform object detection, using target detection YOLO or Mask R-CNN to identify objects in the image;

[0088] The video processing module extracts video frames and selects key frames from the video for analysis. It also performs video encoding and uses 3D CNN or LSTM to extract features of video frames. It also performs speech-to-text conversion and uses speech recognition to convert speech in the video into text.

[0089] The cross-modal fusion module combines the representation of different modalities into a unified representation space using weighted average and attention mechanism methods. It also performs modal fusion and determines the best fusion strategy through experiments.

[0090] Retrieval engine module, which performs similarity calculation through the retrieval engine module, uses cosine similarity and Jaccard similarity methods to calculate the similarity between the query vector and the document vector, and sorts them. Specifically, the results are sorted according to indicators such as similarity scores and document relevance;

[0091] User interaction module, through which the query interface is provided, a simple and easy-to-use user interface is provided, and the user can input text query, upload images or videos;

[0092] The backend service module performs distributed storage: uses a distributed file system (such as Hadoop HDFS) to store large-scale archival data; and deploys it on a cloud platform through cloud services.

[0093] Furthermore, the present invention provides an archive data retrieval device, comprising a memory for storing computer executable instructions; and a processor for implementing any of the above-mentioned document retrieval methods when executing the computer executable instructions stored in the memory.

[0094] Compared with the prior art, the present invention has the following beneficial effects:

[0095] 1. It can more accurately understand the user's query intent and the content of the document, and keyword extraction and sentiment analysis can help the system analyze the document more comprehensively, thereby improving the relevance of the search results. And enhance the search speed. Distributed storage and cloud computing technology can significantly improve the speed of data processing, making large-scale archive retrieval faster. Efficient index structure and parallel processing capabilities help speed up the search process. Through the cross-modal fusion module, the system can process and retrieve multiple types of data including text, images, videos, etc. It allows users to query related data in other modalities through data in any modality. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It is understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0097] Figure 1 is a flow chart of the method of the present invention;

[0098] Figure 2 It is a system structure diagram of the present invention;

[0099] Figure 3 It is the present invention that fuses the representation of text, image and video into a unified vector space graph;

[0100] Figure 4 It is a code screenshot of the data processing of the present invention;

[0101] Figure 5 It is a code screenshot of the text, image and video encoding of the present invention;

[0102] Figure 6 It is a screenshot of the code for the joint similarity calculation of the present invention;

[0103] Figure 7 It is a code screenshot of text sentiment analysis of the present invention;

[0104] Figure 8 It is a specific embodiment of the present invention for the text 1 joint vector calculation diagram;

[0105] Fig. 9 It is a specific embodiment of the present invention for calculating the joint vector of text 2;

[0106] Fig.10 It is a calculation diagram of a specific embodiment of the present invention using cosine similarity to measure similarity text 1;

[0107] Fig.11It is a calculation diagram of a specific embodiment of the present invention using cosine similarity to measure the similarity of text 2. DETAILED DESCRIPTION

[0108] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but is only for selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0109] See also Figure 1-Figure 11 ,The present invention provides an archive data retrieval method S1: firstly, multiple archive data in the system are cleaned and standardized, and the document data is preprocessed, wherein the document data processing specifically includes text, picture and video data;

[0110] Please follow the steps below:

[0111] S1.1: Perform data cleaning, remove noise, correct spelling errors, unify date and time formats, and denoise as shown in formula (1);

[0112]

[0113] Among them, y i is the i-th element in the new sequence after moving average processing; n is the window size of the moving average, the number of data points for calculating the average value;

[0114] x j is the jth element in the original data sequence;

[0115] j=i-(n-1) / 2 to i+(n-1) / 2 is the range of data points involved in the average calculation;

[0116] Specifically, the Levenshtein distance is used to calculate the difference between two strings to correct spelling errors, and the closest correct spelling is applied; as shown in formula (2);

[0117]

[0118] Where D represents the Levenshtein distance function and V is the vocabulary;

[0119] S1.2: All document data have the same metadata format; standardize the data, and specifically use a standardization method to remove, such as formula (3);

[0120]

[0121] Where x is the original data point, μ is the mean, and σ is the standard deviation;

[0122] S1.3: Normalize the data date and convert all dates into a unified format using ISO 8601, as shown in formula (4);

[0123] Normalized Date=ConvertToISO8601(Original Date) Formula (4)

[0124] S2: semantically represent the processed archival data, specifically using a deep learning model to create a semantic vector representation for the document;

[0125] To perform semantic representation, please follow the steps below:

[0126] S2.1: Text encoding uses the Transformer model, BERT or its variants to obtain the semantic representation of the text; specifically, as shown in formula (5);

[0127]

[0128] Among them, t i is a text sequence T = [t1, t2, ..., t n ], use the BERT model to obtain the embedding vector e of each word i ;

[0129] S2.2: Encode the image and use the convolutional neural network (CNN) to extract image features; use the pre-trained ResNet model to encode the image I and obtain the feature vector v I , specifically as formula (6);

[0130] v I =ResNet(I) Formula (6)

[0131] S2.3: Perform video encoding, use 3DCNN or LSTM to extract features of video frames, use the 3DCNN model to encode frames in the video V, and then average the feature vectors of all frames; as shown in formula (7);

[0132]

[0133] Among them, f jis the jth frame in the video, and 3DCNN(·) is the forward propagation function of the 3DCNN model.

[0134] like Figure 2 ,S3: fuse the representations of text, images and videos into a unified vector space, and integrate data of multiple data types such as text, images and videos for cross-modal retrieval;

[0135] Please follow the steps below:

[0136] S3.1: Create a joint table by weighted averaging to fuse the representations of text, image, and video into a unified vector space; as shown in formula (8);

[0137] v joint =αv T +βv I +γv V Formula (8)

[0138] Among them, α, β, and γ are weight coefficients;

[0139] S3.2: The user query is encoded by the encoder in step S3.1; as shown in formula (9);

[0140] q=Encoder(Q) Formula (9)

[0141] Where Q is the text query data provided by the user; Encoder(·) is the encoder function;

[0142] S3.3: Use cosine similarity to measure the joint representation of query vector q and document v joint The similarity between them; as shown in formula (10);

[0143]

[0144] Among them, q·v joint is the product of vectors, ||q|| and ||v joint || are the Euclidean norms of the vectors respectively;

[0145] S3.4: Output the search results according to the similarity ranking.

[0146] S4: Output the search results based on similarity ranking.

[0147] In this embodiment, the text processing module also performs sentiment analysis on the text content of the document to identify the sentiment tendency of the document, which is specifically performed in the following steps:

[0148] S5.1: Use TF-IDF to convert the text into a numerical vector; and calculate the term frequency-inverse document frequency, as shown in equations (11)-(12);

[0149] TF-IDF(t,d,D)=TF(t,d)×IDF(t,D) Equation (11)

[0150] Where TF(t, d) is the frequency of word r in document d, and IDF(t, D) is the frequency of word t in the inverse document;

[0151]

[0152] Where N is the total number of documents, n t is the number of documents containing word t;

[0153] S5.2: Calculate the sentiment score of the document and perform sentiment classification, as shown in formula (13)-formula (14);

[0154] y=σ(w T x+b) Formula (13)

[0155] Among them, x is the feature quantity, w is the weight vector, b is the bias term, and σ(·) is the sigmoid function;

[0156]

[0157] Where n is the number of words in the document, w i For a given word t i The weight of TF-IDF(t i , d, D) is the word t i TF-IDF value in document d.

[0158] In this embodiment, the present invention further includes the following method for cross-modal retrieval:

[0159] S6.1: First, construct an index structure for the processed data, where the index structure is a KD tree structure; the index structure is constructed based on recursive partitioning, each time a dimension is selected and the median is found on the dimension as the split point; the training data set is used as the basic data for constructing the index structure; and the document data identified in step S1 is used to maintain the index, and the real-time data stream continuously updates the index structure.

[0160] S6.2: When a document query is received, the index structure is used to quickly locate the possible candidate set; accurate distance calculations are performed on the elements in the candidate set to find the nearest neighbors. When a query is received, the tree is traversed recursively, and each time the coordinates of the query point are used to decide whether to enter the left subtree or the right subtree. If a leaf node is encountered, all data points in the node are recorded;

[0161] When backtracking, check whether the subtree on the other side may contain better candidate points;

[0162] Calculate the distance between the query point q and the data point p. Specifically, perform accurate distance calculation to find the true nearest neighbor from the candidate set. First, calculate the exact distance between each data point in the candidate set and the query point, and then find the data point with the minimum distance as the nearest neighbor as shown in formula (15);

[0163]

[0164] Among them, p i and q i are the values ​​of data point,p,and query point,q,on the i-th dimension, respectively, and k is the dimension of the data point;

[0165] In this embodiment, the specific examples are as follows, for example, text 1: "A cat sits on the sofa"

[0166] Text 2: "The dog is running on the grass"

[0167] Images: Image 1: a photo of a cat sitting on a sofa, Image 2: a photo of a dog running on the grass

[0168] Videos: Video 1: a video clip of a cat sitting on a sofa, Video 2: a video clip of a dog running on the grass;

[0169] Preprocess the text data, clean the text, remove punctuation, stop words, etc. Encode the text into a vector and use the BERT model. Image: Preprocess the image, such as scaling, cropping, etc. Use the pre-trained ResNet model to encode the image into a vector. Video: Extract the key frames of the video. Use the 3D CNN model to encode the key frames;

[0170] Fuse text, image, and video representations into a unified vector space;

[0171] Text 1:v T1 = [0.1, 0.2, 0.3, 0.4];

[0172] Text 2: v T2 = [0.5, 0.6, 0.7, 0.8];

[0173] Image vector: Image 1: vI1 = [0.15, 0.25, 0.35, 0.45]

[0174] Image 2: v I2 = [0.55, 0.65, 0.75, 0.85];

[0175] Video Vector:

[0176] Video vector 1: v V1 =[0.2, 0.3, 0.4, 0.5]

[0177] Video vector 2: v V2 =[0.6, 0.7, 0.8, 0.9]

[0178] User query: Query: "a cat"

[0179] Query vector: Use the same encoder to convert the query into a vector

[0180] q=[0.12, 0.22, 0.32, 0.42];

[0181] Joint statement

[0182] Joint representation: vectors of different modalities are fused into a unified vector space.

[0183] Use simple weighted average as the fusion strategy, assuming that the weights are α = 0.4, β = 0.3 and γ = 0.3;

[0184] Joint representation vector:

[0185] Document 1: Figure 8

[0186] Document 2: Fig. 9 ;

[0187] Use cosine similarity to measure the joint representation of query vector q and document v joint The similarity between

[0188] Document 1: Fig.10

[0189] Document 2: Fig.11 ;

[0190] Document 1: Similarity1≈9.986

[0191] Document 2: Similarity2≈3.08;

[0192] Document 1 has a higher similarity to the query "a cat".

[0193] In this embodiment, the present invention provides an archive data retrieval system, including a data preprocessing module, wherein the data preprocessing module includes data cleaning, specifically text cleaning, removing irrelevant characters and punctuation marks, performing spelling correction, and data standardization: unifying the format of the document, including date format, uppercase and lowercase;

[0194] Includes a text processing module that performs text encoding and obtains the semantic representation of documents through Transformer, BERT or RoBERTa;

[0195] It also performs keyword extraction, extracts keywords from documents through TF-IDF to enhance retrieval results; and performs sentiment analysis to identify the sentiment tendency of documents;

[0196] Image processing module, which encodes images in documents and extracts image features using a pre-trained CNN model;

[0197] And perform object detection, using target detection YOLO or Mask R-CNN to identify objects in the image;

[0198] The video processing module extracts video frames and selects key frames from the video for analysis. It also performs video encoding and uses 3D CNN or LSTM to extract features of video frames. It also performs speech-to-text conversion and uses speech recognition to convert speech in the video into text.

[0199] The cross-modal fusion module combines the representation of different modalities into a unified representation space using weighted average and attention mechanism methods. It also performs modal fusion and determines the best fusion strategy through experiments.

[0200] Retrieval engine module, which performs similarity calculation through the retrieval engine module, uses cosine similarity and Jaccard similarity methods to calculate the similarity between the query vector and the document vector, and sorts them. Specifically, the results are sorted according to indicators such as similarity scores and document relevance;

[0201] User interaction module, through which the query interface is provided, a simple and easy-to-use user interface is provided, and the user can input text query, upload images or videos;

[0202] The backend service module performs distributed storage: uses a distributed file system (such as Hadoop HDFS) to store large-scale archival data; and deploys it on a cloud platform through cloud services.

[0203] In this embodiment, the present invention provides an archive data retrieval device, including a memory for storing computer executable instructions; and a processor for implementing any of the above-mentioned document retrieval methods when executing the computer executable instructions stored in the memory.

[0204] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention has various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for retrieving archive data, characterized in that: Please follow the steps below: S1: First, multiple archive data in the system are cleaned and standardized, and the document data is pre-processed, where the document data processing specifically includes text, image and video data; S2: Semantically represent the processed archival data, specifically using a deep learning model to create a semantic vector representation for the document; S3: Fusion of text, image, and video representations into a unified vector space, and integration of data from multiple data types such as text, image, and video for cross-modal retrieval; S4: Output the search results according to the similarity ranking.

2. The archival data retrieval method, system and device according to claim 1, characterized in that: In step S1, the following steps are specifically performed: S1.1: Perform data cleaning, remove noise, correct spelling errors, unify date and time formats, and denoise as shown in formula (1); Among them, y i is the i-th element in the new sequence after moving average processing; n is the window size of the moving average, the number of data points for calculating the average value; x j is the jth element in the original data sequence; j=i-(n-1) / 2 to i+(n-1) / 2 is the range of data points involved in the average calculation; Specifically, the Levenshtein distance is used to calculate the difference between two strings to correct spelling errors, and the closest correct spelling is applied; as shown in formula (2); Where D represents the Levenshtein distance function and V is the vocabulary; S1.2: All document data have the same metadata format; standardize the data, and specifically use a standardization method to remove it, such as formula (3); Where x is the original data point, μ is the mean, and σ is the standard deviation; S1.3: Normalize the data date and convert all dates into a unified format using ISO 8601, as shown in formula (4); Normalized Date=ConvertToISO8601(Original Date) Formula (4).

3. The archival data retrieval method according to claim 1, characterized in that: In step S2, semantic representation is performed, which is specifically performed in the following steps: S2.1: Text encoding uses the Transformer model, BERT or its variants to obtain the semantic representation of the text; specifically, as shown in formula (5); Among them, t i is a text sequence T = [t1, t2, ..., t n ], use the BERT model to obtain the embedding vector e of each word i ; S2.2: Encode the image and use the convolutional neural network CNN to extract image features; use the pre-trained ResNet model to encode the image I and obtain the feature vector v I , specifically as formula (6); v I =ResNet(I) Formula (6) S2.3: Perform video encoding, use 3D CNN or LSTM to extract features of video frames, use the 3D CNN model to encode frames in the video V, and then average the feature vectors of all frames; as shown in formula (7); Among them, f j is the jth frame in the video, and 3DCNN(·) is the forward propagation function of the 3D CNN model.

4. The archival data retrieval method according to claim 1, characterized in that: In step S3, the following steps are specifically performed: S3.1: Create a joint table by weighted averaging to fuse the representations of text, image, and video into a unified vector space; as shown in equation (8); v joint =αv T +βv I +γv V expression(8) Among them, α, β, and γ are weight coefficients; S3.2: The user query is encoded by the encoder in step S3.1; as shown in formula (9); q=Encoder(Q) Formula (9) Where Q is the text query data provided by the user; Encoder(·) is the encoder function; S3.3: Use cosine similarity to measure the joint representation of query vector q and document v joint The similarity between them; as shown in formula (10); Among them, q·v joint is the product of vectors, ||q|| and ||v joint || are the Euclidean norms of the vectors respectively; S3.4: Output the search results according to the similarity ranking.

5. The archival data retrieval method according to claim 1, characterized in that: In step S2, the text processing module also performs sentiment analysis on the text content of the document to identify the sentiment tendency of the document, which is specifically performed in the following steps: S5.1: Use TF-IDF to convert the text into a numerical vector; and calculate the term frequency-inverse document frequency, as shown in equations (11)-(12); TF-IDF(t,d,D)=TF(t,d)×IDF(t,D) Equation (11) Among them, TF(t, d) is the frequency of word t in document d, and IDF(t, D) is the frequency of word t in the inverse document; Where N is the total number of documents, n t is the number of documents containing word t; S5.2: Calculate the sentiment score of the document and perform sentiment classification, as shown in formula (13)-formula (14); y=σ(ω T x+b) expression(13) Among them, x is the feature quantity, w is the weight vector, b is the bias term, and σ(·) is the sigmoid function; Where n is the number of words in the document, w i For a given word t i The weight of TF-IDF(t i , d, D) is the TF-IDF value of word t in document d.

6. The archival data retrieval method according to claim 1, characterized in that: In step S3, the cross-modal retrieval further includes the following method: S6.1: First, construct an index structure for the processed data, where the index structure is a KD tree structure; S6.2: When a document query is received, use the index structure to quickly locate the possible candidate set; perform accurate distance calculations on the elements in the candidate set to find the nearest neighbors.

7. The archival data retrieval method according to claim 6, characterized in that: In step S6.1, the index structure is constructed based on recursive partitioning. One dimension is selected each time and the median is found on the dimension as the split point. The training data set is used as the basic data for constructing the index structure. The document data identified in step S1 is used to maintain the index, and the real-time data stream continuously updates the index structure.

8. The archival data retrieval method according to claim 6, characterized in that: In step S6,2, when the query information is received, the tree is traversed recursively, and each time it is decided whether to enter the left subtree or the right subtree according to the coordinates of the query point. If a leaf node is encountered, all data points in the node are recorded; When backtracking, check whether the subtree on the other side may contain better candidate points; Calculate the distance between the query point q and the data point p. Specifically, perform accurate distance calculation to find the true nearest neighbor from the candidate set. First, calculate the exact distance between each data point in the candidate set and the query point, and then find the data point with the minimum distance as the nearest neighbor as shown in formula (15); Among them, p i and q i are the values ​​of the data point, p, and the query point q in the i-th dimension, respectively, and k is the dimension of the data point.

9. An archive data retrieval system, characterized in that: It includes a data preprocessing module, which includes data cleaning, specifically text cleaning, removing irrelevant characters and punctuation marks, performing spelling correction, and data standardization: unifying the format of the document, including date format, uppercase and lowercase; Includes a text processing module that performs text encoding and obtains the semantic representation of documents through Transformer, BERT or RoBERTa; It also performs keyword extraction, extracts keywords from documents through TF-IDF to enhance retrieval results; and performs sentiment analysis to identify the sentiment tendency of documents; Image processing module, which encodes images in documents and extracts image features using a pre-trained CNN model; And perform object detection, using target detection YOLO or Mask R-CNN to identify objects in the image; Video processing module, through which video frames are extracted, key frames are selected from the video for analysis; video encoding is performed, and 3D CNN or LSTM is used to extract features of video frames; And perform speech-to-text conversion, using speech recognition to convert the speech in the video into text; The cross-modal fusion module combines the features of different modalities into a unified representation space using weighted average and attention mechanism methods. And perform modal fusion and determine the best fusion strategy through experiments; Retrieval engine module, which performs similarity calculation through the retrieval engine module, uses cosine similarity and Jaccard similarity methods to calculate the similarity between the query vector and the document vector, and sorts them. Specifically, the results are sorted according to indicators such as similarity scores and document relevance; User interaction module, through which the query interface is provided, a simple and easy-to-use user interface is provided, and the user can input text query, upload images or videos; The back-end service module performs distributed storage: uses a distributed file system to store large-scale archival data; and deploys it on a cloud platform through cloud services.

10. An archive data retrieval device, characterized in that: It comprises a memory for storing computer executable instructions; and a processor for implementing the document retrieval method according to any one of claims 1 to 8 when executing the computer executable instructions stored in the memory.

Images