Instance library optimization method, system and storage medium in digital archive retrieval

Abstract

The application provides an example library optimization method and system in digital archive retrieval and a storage medium. The conversion result of structured data and unstructured data is used for retrieval deviation probability prediction, the retrieval optimization of the digital archives of colleges and universities is pushed forward from the retrieval link to the unstructured data conversion link, data source problems in retrieval can be found early, the conversion result of the unstructured data is optimized, the possibility of losing data background and implicit relationship of the unstructured data in the structured processing process is reduced, the data external structure characteristics and internal semantic characteristics are deeply fused, structured data for retrieval is formed, the archive retrieval plan is dynamically adjusted, and rich semantic information, value information and the association relationship between archive data are provided for archive retrieval.

Description

Technical Field

[0001] This invention relates to the field of digital archive data analysis technology, and in particular to a method, system, and storage medium for optimizing instance databases in digital archive retrieval. Background Technology

[0002] University archives are historical records in various formats, including text, charts, and audio-visual materials, created during the management and operation of universities. Their core content includes aspects such as the university's development history, teaching, research, party and government administration, and student affairs.

[0003] With the continuous development of computer technology, scanning technology, OCR technology, digital photography technology (including audio and video recording), database technology, multimedia technology, and storage technology, traditional university archives are transforming into a new form of archival information—digital archives. This transformation converts archival resources from various media into digital form, connects them through networks, and utilizes computer systems for efficient management. This provides universities with objective and authentic information, which is helpful for major university decisions and important deployments.

[0004] However, the core of existing intelligent retrieval systems for university archives mainly involves extracting structured data to form instances, storing these instances in an instance library, and then using technologies such as resource description frameworks and ontology-based logical reasoning to construct a domain ontology data knowledge base. This process establishes relationships between data and creates indexes to facilitate the system's understanding of the overall situation within the domain and to enable targeted knowledge retrieval.

[0005] In fact, the vast amount of semantically rich unstructured data generated in university teaching, research, and management activities constitutes the main component of university archives data, far exceeding the structured data in the relational databases of management information systems. For this unstructured data, primarily in the form of documents, images, and audio / video, existing intelligent retrieval systems for university archives merely extract information and transform it into structured data through conventional methods such as feature recognition, information extraction, and format transformation, describing it in a structured form, and then extracting instances. However, the retrieval effect based on the formed instance database cannot be controlled, nor can early intelligent optimization of the retrieval system be achieved. Furthermore, the rich semantic information of unstructured data is not effectively utilized in existing intelligent retrieval systems for university archives due to this simple processing, resulting in low retrieval performance. Therefore, it is necessary to provide a new approach to solve the above-mentioned technical problems. Summary of the Invention

[0006] To achieve the above-mentioned objectives and other advantages of the present invention, a first objective of the present invention is to provide a method for optimizing instance databases in digital archive retrieval, comprising the following steps:

[0007] The archival data to be processed is preprocessed to obtain structured and unstructured data;

[0008] The structured data is discretized to obtain discrete features;

[0009] Based on a globally discretized vocabulary, the discrete features are converted into structured data feature vectors;

[0010] The unstructured data is processed by combining the preset archive entity classification information to obtain the unstructured data feature vector;

[0011] The structured data feature vector and the unstructured data feature vector are fused to obtain the fused feature.

[0012] The fused features are input into the instance library evaluation model to calculate and predict the retrieval offset probability based on the current instance library;

[0013] The transformation results of the unstructured data are optimized based on the retrieval offset probability to obtain an optimized instance library.

[0014] Furthermore, the preprocessing of the archive data to be processed includes:

[0015] The archive data to be processed is cleaned, converted in format, and normalized.

[0016] Furthermore, the step of discretizing the structured data to obtain discrete features includes:

[0017] The features of the structured data are converted into discrete features using a tree-building algorithm.

[0018] Furthermore, the step of converting the discrete features into structured data feature vectors based on a globally discretized vocabulary includes:

[0019] The discrete features share the global discretized vocabulary;

[0020] Assign an index value to each discrete feature;

[0021] The index values ​​are mapped to vectors through a learning process and then concatenated to obtain structured data feature vectors.

[0022] Furthermore, the step of processing the unstructured data by combining preset archive entity classification information to obtain the unstructured data feature vector includes:

[0023] Obtain the preset file entity classification information;

[0024] The unstructured data is segmented based on the preset archive entity classification information to obtain the segmentation results;

[0025] The word segmentation results are used as input to a pre-trained word vector model to generate dynamic word vectors;

[0026] Pooling is performed on each column of the dynamic word vector to compress the dynamic word vector into a first semantic vector.

[0027] Furthermore, the pre-trained word vector model includes an input layer, an encoder, a prediction layer, and an output layer; wherein,

[0028] The input layer is used to represent the input word segmentation results as word vectors;

[0029] The encoder contains multiple layers of Transformer encoders, which are used to capture the contextual relationships in the input sequence and encode them into fixed-length vector representations. Each Transformer encoder includes a self-attention layer and a fully connected feedforward layer for feature extraction and representation learning of the input sequence.

[0030] The prediction layer is used to randomly mask several identifiers in the input sequence through a self-attention mechanism, and to predict the content of the masked identifiers using the unmasked words.

[0031] The output layer is used to represent the input sequence as a vector to obtain dynamic word vectors.

[0032] Furthermore, the step of processing the unstructured data by combining preset archive entity classification information to obtain the unstructured data feature vector also includes:

[0033] Archive entities are extracted from the unstructured data using an archive entity extraction model to obtain an archive entity set.

[0034] Perform a completion operation on the set of archive entities whose number of archive entities does not reach the entity threshold;

[0035] Each set of archive entities is input into the pre-trained word vector model to obtain a second semantic vector;

[0036] Perform a dimensionality reduction operation on the second semantic vector so that the dimension of the second semantic vector is the same as the dimension of the first semantic vector;

[0037] The second semantic vector is combined with the first semantic vector to obtain the unstructured data feature vector.

[0038] Further, the step of fusing the structured data feature vector with the unstructured data feature vector to obtain the fused feature includes:

[0039] The structured data feature vector and the unstructured data feature vector are fused using a probabilistic sparse self-attention mechanism.

[0040] Furthermore, the step of using a probabilistic sparse self-attention mechanism to perform feature fusion on the structured data feature vector and the unstructured data feature vector includes:

[0041] For each input element, for a given query and key, calculate the dot product of the query and the key to obtain the attention score;

[0042] The attention scores are filtered using a probabilistic method, retaining only those scores that meet the importance requirements;

[0043] The selected attention scores are normalized, and attention weights are calculated.

[0044] The final output is obtained by weighting and summing the attention weights.

[0045] Furthermore, the instance library evaluation model is a neural network model based on the Sigmoid function, and the neural network model uses the cross-entropy loss function;

[0046] Before inputting the fused features into the instance library evaluation model for prediction, the following steps are also included:

[0047] The fusion features are subjected to residual and layer normalization operations.

[0048] Further, the step of optimizing the transformation result of the unstructured data based on the retrieval offset probability to obtain the optimized instance library includes:

[0049] When the retrieval offset probability reaches the probability threshold, the unstructured data feature vector and the latest conversion result of the unstructured data conversion model are used as inputs to the unstructured data conversion model to obtain the current conversion result.

[0050] The current transformation result is used as the input at the current timestamp of the long short-term memory neural network model. The hidden state of the previous timestamp and the cell state of the previous time step are combined to transform the data, thereby obtaining the current transformation result of the unstructured data, as well as the cell state and the hidden state at the current time step.

[0051] The current transformation result of the unstructured data is combined with the structured data to obtain the optimized instance library.

[0052] The second objective of this invention is to provide a system for optimizing a database of instances in digital archive retrieval. This system implements the aforementioned method and includes a data preprocessing module, a structured data discretization module, a structured data feature vector generation module, an unstructured data feature vector generation module, a feature fusion module, a retrieval offset prediction module, and a database optimization module.

[0053] The archival data preprocessing module is used to preprocess the archival data to be processed to obtain structured data and unstructured data.

[0054] The structured data discretization processing module is used to discretize the structured data to obtain discrete features;

[0055] The structured data feature vector generation module is used to convert the discrete features into structured data feature vectors based on a global discrete vocabulary.

[0056] The unstructured data feature vector generation module is used to process the unstructured data in conjunction with preset archive entity classification information to obtain unstructured data feature vectors.

[0057] The feature fusion module is used to fuse the structured data feature vector and the unstructured data feature vector to obtain fused features;

[0058] The retrieval offset prediction module is used to input the fused features into the instance library evaluation model and calculate the predicted retrieval offset probability based on the current instance library.

[0059] The instance library optimization module is used to optimize the transformation result of the unstructured data according to the retrieval offset probability to obtain an optimized instance library.

[0060] A third objective of the present invention is to provide a computer-readable storage medium having program instructions stored thereon, which, when executed, implement the method described above.

[0061] Compared with the prior art, the beneficial effects of the present invention are:

[0062] This invention provides a method, system, and storage medium for optimizing instance libraries in digital archive retrieval. By using the conversion results of structured and unstructured data for retrieval offset probability prediction, it advances the optimization of retrieval of digital archives in universities from the retrieval stage to the unstructured data conversion stage. This allows for earlier detection of data source problems, optimization of unstructured data conversion results, and reduction of the possibility of losing data background and implicit relationships during the structuring process. It also deeply integrates the external structural features and internal semantic features of the data to form retrieval-ready structured data, dynamically adjusting the archive retrieval plan and providing rich semantic information, value information, and relationships between archive data for archive retrieval.

[0063] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it according to the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Specific embodiments of the present invention are given in detail below with reference to the accompanying drawings. Attached Figure Description

[0064] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0065] Figure 1 This is a flowchart of the instance library optimization method in digital archive retrieval in Example 1;

[0066] Figure 2 This is a flowchart of the structured data feature vector generation process in Example 1;

[0067] Figure 3 The process for generating feature vectors from unstructured data in Example 1 Figure 1 ;

[0068] Figure 4 The process for generating feature vectors from unstructured data in Example 1 Figure 2 ;

[0069] Figure 5 This is a flowchart of the feature fusion process in Example 1;

[0070] Figure 6 The flowchart for optimizing the transformation results of unstructured data in Example 1;

[0071] Figure 7 This is a schematic diagram of the instance database optimization system in the digital archive retrieval of Example 2;

[0072] Figure 8 This is a schematic diagram of the storage medium in Example 3. Detailed Implementation

[0073] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0074] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.

[0075] The drawing numbers in this application are only used to distinguish the steps in the scheme and are not used to limit the execution order of the steps. The specific execution order is as described in the specification.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Example 1

[0077] An example database optimization method for digital archive retrieval, such as Figure 1 As shown, it includes the following steps:

[0078] S1. Preprocess the archive data to be processed to obtain structured data and unstructured data;

[0079] Unstructured data in university digitized archives refers to data that cannot be stored and retrieved using traditional database tables, including text, images, audio, and video. Unlike structured data (such as tabular data in a database), unstructured data does not have a fixed format; its content and structure are usually determined by the natural form of the data.

[0080] University archives generate massive amounts of data daily, including student records, faculty records, and school management data. This unstructured big data in university digitization archives primarily comprises student records, faculty records, school management data, and other sources. Student records include student registration forms, transcripts, and student status changes; faculty records include resumes, teaching achievements, and research results; school management data includes school notices, documents, and policies; and other sources include library borrowing records and campus card information. Furthermore, university archives involve a wide variety of data types, including text, images, audio, and video. Therefore, this unstructured data is characterized by diversity, complexity, volume, and rapid change. University archive data has high value and can be used for student management, faculty evaluation, and school decision-making.

[0081] In some embodiments, the preprocessing of the archive data to be processed includes:

[0082] Cleaning, format conversion, and data normalization are performed on the archive data to be processed.

[0083] Unstructured data usually contains a large amount of noise, repetition, and error information. Through data cleaning, these unnecessary parts can be removed to ensure the accuracy and consistency of the data. The steps of data cleaning include data deduplication, filling missing values, correcting data errors, and standardizing data formats. For example, when processing text data, HTML tags, special characters, and stop words (such as "of", "is", etc.) can be removed. The result of data cleaning is a cleaner and more orderly dataset, laying a foundation for subsequent data conversion and analysis.

[0084] Format conversion is to convert data in different formats (such as text, images, audio) into a unified format for convenient subsequent processing. Data normalization is to scale the data to a specific range to eliminate the dimensional difference between different features.

[0085] S2. Perform discretization processing on the structured data to obtain discrete features;

[0086] In some embodiments, the step of performing discretization processing on the structured data to obtain discrete features includes:

[0087] Convert the features of the structured data into discrete features through a tree construction algorithm. For example, the CART algorithm can handle discrete problems and continuous problems. When dealing with continuous problems, binary splitting is mainly used to handle continuous variables, that is, if the feature value is greater than a given value, it goes to the left subtree, otherwise it goes to the right subtree. Converting the features of structured data into discrete features reduces the complexity of the data while introducing data semantic information. The converted discrete features can not only express data semantics but also unified the feature form.

[0088] S3. Based on the global discretization vocabulary, convert the discrete features into structured data feature vectors;

[0089] Furthermore, as Figure 2 shown, the step of converting the discrete features into structured data feature vectors based on the global discretization vocabulary includes:

[0090] S31. Let the discrete features share the global discretization vocabulary;

[0091] The global discretized vocabulary is obtained by selecting features from the data to be processed using the Apriori algorithm, constructing a core vocabulary and a dependency vocabulary using the pre-words and successor words in the mined features, and filtering out candidate sets with low semantic relevance.

[0092] By using a globally discretized vocabulary, different discrete features can be modeled into the same discrete, fine-grained representation.

[0093] S32. Assign index values ​​to the different values ​​of each discrete feature;

[0094] S33. The index values ​​are mapped to vectors through the learning process and concatenated to obtain structured data feature vectors.

[0095] By converting discrete features into semantic vectors through a global discretization vocabulary, not only is the semantic information expanded, but semantic relationships are also created between different discrete features.

[0096] S4. Process the unstructured data by combining the preset archive entity classification information to obtain the unstructured data feature vector;

[0097] Optionally, the pre-defined classification information for archival entities can be compiled based on past experience and the actual work of university archives. Specifically, the primary categories of the entity classification outline are: Party and Mass Archives, Administrative Archives, Teaching Archives, Scientific Research Archives, Basic Construction Archives, Instrument and Equipment Archives, Product Production and Technological Development Archives, Publication Archives, Financial and Accounting Archives, Personal Performance Archives, and Physical Archives.

[0098] The secondary categories of the entity classification outline are as follows: Party and mass organizations, administrative, publishing, and accounting archives are divided into secondary categories according to their functional divisions; teaching archives are divided into secondary categories according to the content, characteristics, and formation patterns of the documents and materials; scientific research, basic construction, instrumentation and equipment, product production, and technological development archives are divided into secondary categories according to projects; faculty and staff personal performance archives are divided into secondary categories according to their department number (or code); and physical archives are divided into secondary categories according to the type of physical carrier.

[0099] In some embodiments, such as Figure 3 As shown, the step of processing the unstructured data by combining preset archive entity classification information to obtain the unstructured data feature vector includes:

[0100] S41. Obtain preset file entity classification information;

[0101] S42. Perform word segmentation on the unstructured data according to the preset archive entity classification information to obtain the word segmentation result;

[0102] Optionally, a word segmentation tool in Python, such as jieba, can be used to segment unstructured data based on preset file entity classification information, resulting in a set consisting of multiple entities, conjunctions, punctuation marks, etc.

[0103] S43. Use the word segmentation results as input to a pre-trained word vector model to generate dynamic word vectors;

[0104] S44. Perform pooling operation on each column of the dynamic word vector to compress the dynamic word vector into a first semantic vector.

[0105] Pooling operations can be used to integrate the semantic information of dynamic word vectors. For example, average pooling, max pooling, random pooling, and global average pooling can be performed on each column of the dynamic word vectors. Pooling can reduce the size of the feature map, which means reducing the amount of computation and the required GPU memory. Taking the average pooling operation on each column of the dynamic word vectors as an example, only the average of the feature points in the neighborhood needs to be calculated.

[0106] In some embodiments, the pre-trained word vector model includes an input layer, an encoder, a prediction layer, and an output layer; wherein,

[0107] The input layer is used to represent the input word segmentation results as word vectors;

[0108] The encoder contains multiple layers of Transformer encoders, which are used to capture the contextual relationships in the input sequence and encode them into fixed-length vector representations. Each Transformer encoder includes a self-attention layer and a fully connected feedforward layer for feature extraction and representation learning of the input sequence.

[0109] The prediction layer is used to randomly mask several identifiers in the input sequence through a self-attention mechanism, and to predict the content of the masked identifiers using the unmasked words.

[0110] The output layer is used to represent the input sequence as a vector to obtain dynamic word vectors.

[0111] In some embodiments, such as Figure 4 As shown, the step of processing the unstructured data by combining preset archive entity classification information to obtain the unstructured data feature vector further includes:

[0112] S45. Extract archive entities from the unstructured data using the archive entity extraction model to obtain an archive entity set;

[0113] Optionally, the archival entity extraction model can be trained using a pre-trained natural language processing model based on a university-specific dataset; the pre-trained natural language processing model can be a model such as BERT. BERT employs a Transformer architecture, specifically implemented through multiple layers of Transformer encoders.

[0114] S46. Perform a completion operation on the set of archive entities whose number of archive entities has not reached the entity threshold;

[0115] Because the number of entities contained in a set of archive entities may vary, in order to ensure consistency in subsequent processing, special characters can be added to the beginning or end of the archive entity set whose number of entities has not reached the entity threshold, so that the number of entities reaches the entity threshold.

[0116] S47. Input each set of archive entities into the pre-trained word vector model to obtain the second semantic vector;

[0117] S48. Perform a dimensionality reduction operation on the second semantic vector so that the dimension of the second semantic vector is the same as the dimension of the first semantic vector;

[0118] The second semantic vector is reduced in dimensionality to facilitate subsequent fusion with the feature vector of structured data.

[0119] Optionally, the second semantic vector can be reduced in dimensionality through a fully connected layer. The fully connected layer flattens the output data of the convolutional layer, which greatly reduces the dimensionality of the data and thus reduces the computational burden of the fully connected layer.

[0120] S49. Combine the second semantic vector with the first semantic vector to obtain an unstructured data feature vector.

[0121] S5. The structured data feature vector and the unstructured data feature vector are fused to obtain fused features;

[0122] In some embodiments, the step of fusing the structured data feature vector with the unstructured data feature vector to obtain fused features includes:

[0123] The structured data feature vector and the unstructured data feature vector are fused using a probabilistic sparse self-attention mechanism.

[0124] Sparse self-attention is an improved self-attention mechanism designed to enhance computational efficiency and reduce computational complexity. In traditional self-attention mechanisms, each input element requires calculating attention weights with all other input elements, resulting in high computational complexity. This complexity becomes extremely high when the sequence length is large, making it difficult to handle long sequences.

[0125] Sparse self-attention reduces computational complexity by introducing a sparse matrix, allowing each input element to calculate attention weights only with a subset of input elements.

[0126] Specifically, such as Figure 5 As shown, the step of using a probabilistic sparse self-attention mechanism to perform feature fusion on the structured data feature vector and the unstructured data feature vector includes:

[0127] S51. For each input element, for a given query and key, calculate the dot product of the query and the key to obtain the attention score;

[0128] S52. The attention scores are filtered using a probability method, retaining only the scores that meet the importance requirements; for example, the top k largest values ​​can be selected, or the scores can be filtered according to a certain threshold.

[0129] S53. Normalize the filtered attention scores and calculate the attention weights;

[0130] S54. Use the attention weights to perform a weighted summation to obtain the final output.

[0131] S6. Input the fused features into the instance library evaluation model and calculate the predicted retrieval offset probability based on the current instance library;

[0132] By predicting and evaluating the retrieval performance of fusion features, the optimization of retrieval of digital archives of universities can be pushed forward from the retrieval stage to the unstructured data transformation stage. This allows for the early detection of data source problems, optimization of unstructured data transformation results, and reduction of the possibility of losing data background and implicit relationships during the structuring process.

[0133] In some embodiments, the instance library evaluation model is a neural network model based on the Sigmoid function, and the neural network model uses the cross-entropy loss function;

[0134] Alternatively, the neural network model can be a feedforward neural network model, in which data passes through different input nodes until it reaches the output node. In contrast to backpropagation, the data here moves in only one direction.

[0135] In some embodiments, the method further includes the following before inputting the fused features into the instance library evaluation model for prediction:

[0136] The fused features are subjected to residual and layer normalization operations to solve the gradient vanishing problem.

[0137] S7. Optimize the transformation result of the unstructured data according to the retrieval offset probability to obtain the optimized instance library.

[0138] By optimizing the transformation results of unstructured data and deeply integrating the external structural features and internal semantic features of the data to form retrieval-available structured data, the role of archival retrieval analysis can be brought into play. This allows for dynamic adjustment of archival retrieval planning and provides rich semantic information, value information, and relationships between archival data for archival retrieval.

[0139] In some embodiments, such as Figure 6 As shown, the step of optimizing the transformation result of the unstructured data based on the retrieval offset probability to obtain the optimized instance library includes:

[0140] S71. When the retrieval offset probability reaches the probability threshold, the unstructured data feature vector and the latest conversion result of the unstructured data conversion model are used as inputs to the unstructured data conversion model to obtain the current conversion result.

[0141] Unstructured data is characterized by diversity, complexity, large volume, and rapid change. The transformation results of unstructured data can be optimized based on previous transformations to achieve efficient transformation. Optionally, the unstructured data transformation model can employ the Informer model, which is an optimization of the classic Transformer model.

[0142] S72. The current transformation result is used as the input at the current timestamp of the long short-term memory neural network model. The hidden state of the previous timestamp and the unit state of the previous time are combined to transform the data to obtain the current transformation result of the unstructured data, as well as the unit state and the hidden state at the current time.

[0143] Long Short-Term Memory (LSTM) neural network models only need to process one multi-dimensional feature at each time step, which greatly reduces the amount of data processed at each time step and thus reduces computation time.

[0144] S73. Combine the current transformation result of the unstructured data with the structured data to obtain the optimized instance library.

[0145] This embodiment provides a method for optimizing the instance library in digital archive retrieval. By using the conversion results of structured and unstructured data for retrieval offset probability prediction, the optimization of retrieval of digital archives in universities is pushed forward from the retrieval stage to the unstructured data conversion stage. This allows for the early detection of data source problems, optimization of the conversion results of unstructured data, and reduction of the possibility of losing data background and implicit relationships during the structuring process. It also deeply integrates the external structural features and internal semantic features of the data to form retrieval-ready structured data, thereby dynamically adjusting the archive retrieval plan and providing rich semantic information, value information, and correlations between archive data for archive retrieval. Example 2

[0146] Based on the same concept, this embodiment also provides an instance library optimization system for digital archive retrieval, which applies the instance library optimization method for digital archive retrieval provided in Embodiment 1. For a detailed description of the instance library optimization method for digital archive retrieval provided in Embodiment 1, please refer to the corresponding description in Embodiment 1, which will not be repeated here.

[0147] It is understood that the instance library optimization system for digital archive retrieval provided in this embodiment includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. Combining the units and algorithm steps of the examples disclosed in this embodiment, this embodiment can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solution of this embodiment.

[0148] An instance database optimization system for digital archive retrieval, such as Figure 7 As shown, it includes an archival data preprocessing module 800, a structured data discretization processing module 810, a structured data feature vector generation module 820, an unstructured data feature vector generation module 830, a feature fusion module 840, a retrieval offset prediction module 850, and an instance library optimization module 860; among which,

[0149] The archival data preprocessing module is used to preprocess the archival data to be processed to obtain structured data and unstructured data.

[0150] The structured data discretization processing module is used to discretize the structured data to obtain discrete features;

[0151] The structured data feature vector generation module is used to convert the discrete features into structured data feature vectors based on a global discrete vocabulary.

[0152] The unstructured data feature vector generation module is used to process the unstructured data in conjunction with preset archive entity classification information to obtain unstructured data feature vectors.

[0153] The feature fusion module is used to fuse the structured data feature vector and the unstructured data feature vector to obtain fused features;

[0154] The retrieval offset prediction module is used to input the fused features into the instance library evaluation model and calculate the predicted retrieval offset probability based on the current instance library.

[0155] The instance library optimization module is used to optimize the transformation result of the unstructured data according to the retrieval offset probability to obtain an optimized instance library.

[0156] Based on the technical solutions of the above embodiments, optionally, the preprocessing of the archive data to be processed includes:

[0157] The archive data to be processed is cleaned, converted in format, and normalized.

[0158] Based on the technical solutions of the above embodiments, optionally, the step of discretizing the structured data to obtain discrete features includes:

[0159] The features of the structured data are converted into discrete features using a tree-building algorithm.

[0160] Based on the technical solutions of the above embodiments, optionally, the step of converting the discrete features into structured data feature vectors based on a global discretized vocabulary includes:

[0161] The discrete features share the global discretized vocabulary;

[0162] Assign an index value to each discrete feature;

[0163] The index values ​​are mapped to vectors through a learning process and then concatenated to obtain structured data feature vectors.

[0164] Based on the technical solution of the above embodiments, optionally, the step of processing the unstructured data by combining preset archive entity classification information to obtain the unstructured data feature vector includes:

[0165] Obtain the preset file entity classification information;

[0166] The unstructured data is segmented based on the preset archive entity classification information to obtain the segmentation results;

[0167] The word segmentation results are used as input to a pre-trained word vector model to generate dynamic word vectors;

[0168] Pooling is performed on each column of the dynamic word vector to compress the dynamic word vector into a first semantic vector.

[0169] Based on the technical solutions of the above embodiments, optionally, the pre-trained word vector model includes an input layer, an encoder, a prediction layer, and an output layer; wherein,

[0170] The input layer is used to represent the input word segmentation results as word vectors;

[0171] The encoder contains multiple layers of Transformer encoders, which are used to capture the contextual relationships in the input sequence and encode them into fixed-length vector representations. Each Transformer encoder includes a self-attention layer and a fully connected feedforward layer for feature extraction and representation learning of the input sequence.

[0172] The prediction layer is used to randomly mask several identifiers in the input sequence through a self-attention mechanism, and to predict the content of the masked identifiers using the unmasked words.

[0173] The output layer is used to represent the input sequence as a vector to obtain dynamic word vectors.

[0174] Based on the technical solution of the above embodiments, optionally, the step of processing the unstructured data by combining preset archive entity classification information to obtain the unstructured data feature vector further includes:

[0175] Archive entities are extracted from the unstructured data using an archive entity extraction model to obtain an archive entity set.

[0176] Perform a completion operation on the set of archive entities whose number of archive entities does not reach the entity threshold;

[0177] Each set of archive entities is input into the pre-trained word vector model to obtain a second semantic vector;

[0178] Perform a dimensionality reduction operation on the second semantic vector so that the dimension of the second semantic vector is the same as the dimension of the first semantic vector;

[0179] The second semantic vector is combined with the first semantic vector to obtain the unstructured data feature vector.

[0180] Based on the technical solutions of the above embodiments, optionally, the step of fusing the structured data feature vector and the unstructured data feature vector to obtain fused features includes:

[0181] The structured data feature vector and the unstructured data feature vector are fused using a probabilistic sparse self-attention mechanism.

[0182] Based on the technical solutions of the above embodiments, optionally, the step of using a probabilistic sparse self-attention mechanism to perform feature fusion of the structured data feature vector and the unstructured data feature vector includes:

[0183] For each input element, for a given query and key, calculate the dot product of the query and the key to obtain the attention score;

[0184] The attention scores are filtered using a probabilistic method, retaining only those scores that meet the importance requirements;

[0185] The selected attention scores are normalized, and attention weights are calculated.

[0186] The final output is obtained by weighting and summing the attention weights.

[0187] Based on the technical solutions of the above embodiments, optionally, the instance library evaluation model is a neural network model based on the Sigmoid function, and the neural network model adopts the cross-entropy loss function;

[0188] Before inputting the fused features into the instance library evaluation model for prediction, the following steps are also included:

[0189] The fusion features are subjected to residual and layer normalization operations.

[0190] Based on the technical solution of the above embodiments, optionally, the step of optimizing the transformation result of the unstructured data according to the retrieval offset probability to obtain the optimized instance library includes:

[0191] When the retrieval offset probability reaches the probability threshold, the unstructured data feature vector and the latest conversion result of the unstructured data conversion model are used as inputs to the unstructured data conversion model to obtain the current conversion result.

[0192] The current transformation result is used as the input at the current timestamp of the long short-term memory neural network model. The hidden state of the previous timestamp and the cell state of the previous time step are combined to transform the data, thereby obtaining the current transformation result of the unstructured data, as well as the cell state and the hidden state at the current time step.

[0193] The current transformation result of the unstructured data is combined with the structured data to obtain the optimized instance library.

[0194] This embodiment provides an instance library optimization system for digital archive retrieval. By using the conversion results of structured and unstructured data for retrieval offset probability prediction, the system advances the optimization of retrieval of digital archives in universities from the retrieval stage to the unstructured data conversion stage. This allows for earlier detection of data source problems, optimization of the conversion results of unstructured data, and reduction of the possibility of losing data background and implicit relationships during the structuring process. It also deeply integrates the external structural features and internal semantic features of the data to form retrieval-ready structured data, dynamically adjusting the archive retrieval plan and providing rich semantic information, value information, and correlations between archive data for archive retrieval. Example 3

[0195] A computer-readable storage medium, such as Figure 8 As shown, it stores program instructions, which, when executed, implement a method for optimizing an instance library in digital archive retrieval. For a detailed description of the method, please refer to the corresponding description in the above method embodiments; it will not be repeated here.

[0196] The number of devices and processing scale described herein are for the purpose of simplifying the description of the invention. Applications, modifications, and variations of the invention will be readily apparent to those skilled in the art.

[0197] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

[0198] The apparatus, computer device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, computer device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, computer device, and non-volatile computer storage medium will not be repeated here.

[0199] Those skilled in the art will also know that, besides implementing the controller in the form of purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller take the form of logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices included within it for implementing various functions can also be considered structures within that hardware component. Alternatively, the devices for implementing various functions can be considered as both software units implementing the method and structures within a hardware component.

[0200] The systems, apparatuses, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above apparatuses are described separately as various units based on their functions. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0201] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0202] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0203] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0204] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0205] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0206] This specification may be described in the general context of computer-executable instructions, such as program units, that are executed by a computer. Generally, program units include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program units may reside in local and remote computer storage media, including storage devices.

[0207] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0208] The above description is merely an embodiment of this specification and is not intended to limit the scope of one or more embodiments of this specification. Various modifications and variations can be made to one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. A method for optimizing an instance base in a digital archive search, characterized by The method comprises the following steps: Preprocessing the to-be-processed archive data to obtain structured data and unstructured data; Discretizing the structured data to obtain discrete features; Converting the discrete features into a structured data feature vector based on a global discretization vocabulary; Processing the unstructured data in combination with preset archive entity classification information to obtain an unstructured data feature vector; Fusing the structured data feature vector and the unstructured data feature vector to obtain fused features; Inputting the fused features into an instance library evaluation model to calculate a predicted retrieval offset probability based on a current instance library; Optimizing the conversion result of the unstructured data according to the retrieval offset probability to obtain an optimized instance library; The step of processing the unstructured data in combination with preset archive entity classification information to obtain an unstructured data feature vector comprises: Obtaining preset archive entity classification information; Performing a word segmentation operation on the unstructured data according to the preset archive entity classification information to obtain a word segmentation result; Taking the word segmentation result as input of a pre-trained word vector model to generate a dynamic word vector; Performing a pooling operation on each column in the dynamic word vector to compress the dynamic word vector into a first semantic vector; The pre-trained word vector model comprises an input layer, an encoder, a prediction layer, and an output layer; wherein The input layer is configured to represent the input word segmentation result as a word vector; The encoder comprises multiple layers of Transformer encoders, which are configured to capture the context relationship in the input sequence and encode it into a fixed-length vector representation; each Transformer encoder comprises a self-attention layer and a fully connected feedforward layer, which are configured to perform feature extraction and representation learning on the input sequence; The prediction layer is configured to randomly mask a number of identifiers in the input sequence by using the unmasked words to predict the content of the masked identifiers; The output layer is configured to perform vector representation on the input sequence to obtain a dynamic word vector; The step of optimizing the conversion result of the unstructured data according to the retrieval offset probability to obtain an optimized instance library comprises: When the retrieval offset probability reaches a probability threshold, taking the unstructured data feature vector and the latest conversion result of the unstructured data conversion model as input of the unstructured data conversion model to obtain a current conversion result; Taking the current conversion result as input of a long short-term memory neural network model at a current timestamp, combining the hidden state of the long short-term memory neural network model at a previous timestamp and the cell state at a previous time to obtain the unstructured data current conversion result, the cell state at the current time, and the hidden state at the current time; Combining the unstructured data current conversion result with the structured data to obtain an optimized instance library.

2. The method for optimizing the instance base in digital archive retrieval according to claim 1, wherein, The preprocessing of the to-be-processed archive data comprises: Cleaning, format conversion, and data normalization processing of the to-be-processed archive data.

3. The method for optimizing the instance base in digital archive retrieval according to claim 1, wherein, The step of discretizing the structured data to obtain discrete features comprises: The features of the structured data are converted into discrete features by a tree construction algorithm.

4. The method of claim 3, wherein the method further comprises: - determining the number of instances in the instance library; - determining the number of instances in the instance library that are relevant to the search query; and - determining the number of instances in the instance library that are not relevant to the search query. The step of converting the discrete features into a structured data feature vector based on a global discretization word table comprises: The discrete features share the global discretization word table; Different values of each discrete feature are assigned to index values; The index values are mapped into vectors through a learning process and spliced to obtain a structured data feature vector.

5. The method for optimizing the instance base in digital archive retrieval according to claim 1, wherein, The step of processing the unstructured data based on the preset archive entity classification information to obtain an unstructured data feature vector further comprises: Archive entities are extracted from the unstructured data by an archive entity extraction model to obtain an archive entity set; The archive entity set whose number of archive entities does not reach an entity threshold is subjected to a completion operation; Each archive entity set is input into the pre-trained word vector model to obtain a second semantic vector; The second semantic vector is subjected to a dimension reduction operation so that the dimension of the second semantic vector is the same as that of the first semantic vector; The second semantic vector and the first semantic vector are combined to obtain an unstructured data feature vector.

6. The method of claim 1, wherein the method further comprises: determining a number of instances in the instance library; and determining a number of instances in the instance library that are relevant to the search query. 5 The step of fusing the structured data feature vector and the unstructured data feature vector to obtain a fused feature comprises: The structured data feature vector and the unstructured data feature vector are fused by using a probabilistic sparse self-attention mechanism.

7. A method for optimizing a library of instances in digital archive retrieval as claimed in claim 6, wherein, The step of fusing the structured data feature vector and the unstructured data feature vector by using a probabilistic sparse self-attention mechanism comprises: For each input element, the dot product of a given query and a key is calculated to obtain an attention score; The attention scores are filtered by a probabilistic method, and only scores that meet the importance requirement are retained; The filtered attention scores are subjected to normalization processing, and attention weights are calculated; The attention weights are used for weighted summation to obtain a final output.

8. The method for optimizing the instance database in digital archive retrieval as described in claim 6, characterized in that: The instance library evaluation model is a neural network model based on a Sigmoid function, and the neural network model adopts a cross-entropy loss function; Before the fused feature is input into the instance library evaluation model for prediction, the fused feature is subjected to residual and layer normalization operations. The system comprises an archive data preprocessing module, a structured data discretization processing module, a structured data feature vector generation module, an unstructured data feature vector generation module, a feature fusion module, a retrieval offset prediction module, and an instance library optimization module.

9. A system for optimizing an instance base in a digital archive search, implementing the method according to any one of claims 1 to 8, characterized in that: The archive data preprocessing module is configured to preprocess the archive data to be processed to obtain structured data and unstructured data. The structured data discretization processing module is configured to discretize the structured data to obtain discrete features. The structured data feature vector generation module is configured to convert the discrete features into a structured data feature vector based on a global discretization word table. ​ The unstructured data feature vector generation module is configured to process the unstructured data in combination with preset profile entity classification information to obtain an unstructured data feature vector. The feature fusion module is configured to fuse the structured data feature vector and the unstructured data feature vector to obtain fused features. The retrieval offset prediction module is configured to input the fused features into an instance library evaluation model to calculate a predicted retrieval offset probability based on a current instance library. The instance library optimization module is configured to optimize a conversion result of the unstructured data according to the retrieval offset probability to obtain an optimized instance library.

10. A computer-readable storage medium, characterized in that, A computer readable storage medium has stored thereon program instructions, which when executed implement the method of any one of claims 1-8.

Images