Instance library optimization method and system in digital archive retrieval and storage medium

Abstract

According to the instance library optimization method and system in digital archive retrieval and the storage medium provided by the invention, the conversion result of the structured data and the unstructured data is used for predicting the retrieval offset probability, so that the retrieval optimization of the digital archives of colleges and universities is pushed from a retrieval link to an unstructured data conversion link; according to the method, the problem of a retrieved data source can be found earlier, a conversion result of the unstructured data is optimized, the possibility that the unstructured data loses a data background and an implicit relationship in a structured processing process is reduced, and external structural features and internal semantic features of the data are deeply fused so as to form the retrieved structured data, so that the retrieval efficiency is improved. According to the method, archive retrieval plans are dynamically adjusted, and rich semantic information, value information and association relationships among archive data are provided for archive retrieval.

Description

Technical Field

[0001] The present invention relates to the technical field of digital archive data analysis, and in particular to a case library optimization method, system and storage medium in digital archive retrieval. Background Art

[0002] University archives are historical records in different media such as text, charts, audio and video that have preservation value formed in the management and operation process of universities. Their core content includes the school's development history, teaching, scientific research, party and government management, student affairs and other aspects.

[0003] With the continuous development of computer technology, scanning technology, optical character recognition (OCR) technology, digital photography technology (including audio and video recording), database technology, multimedia technology, and storage technology, traditional university archives have transformed into a new form of archival information: digital archives. This transformation converts archival resources in various media into digital form, connects them through networks, and utilizes computer systems for efficient management. This provides universities with objective and authentic information, facilitating important decision-making and deployment.

[0004] However, the core of the existing intelligent retrieval system for university archives is mainly to extract structured data to form instances, store the instances in the instance library, and then use technologies such as resource description framework and ontology-based logical reasoning to build a domain ontology data knowledge base, build the relationship between data, and establish an index to facilitate the system's understanding of the overall situation in the field and to conduct targeted retrieval of knowledge.

[0005] In fact, the vast amount of semantically rich unstructured data generated in university teaching, scientific research, and management activities is the main component of university archival data. Its data volume far exceeds the structured data in the relational database of the management information system. For this unstructured data, which mainly takes the form of documents, images, audio and video, existing university archive intelligent retrieval systems simply extract information through conventional means such as feature recognition, information extraction, and situation transformation to transform it into structured data, describe it in a structured form, and then extract it to form instances. However, the retrieval effect based on the generated instance library cannot be controlled, and it is impossible to achieve early intelligent optimization of the retrieval system. Moreover, the unstructured data is only processed through this simple method, so its rich semantic information is not effectively utilized in the existing university archive intelligent retrieval system, resulting in low retrieval performance of the existing university archive intelligent retrieval system. Therefore, it is necessary to provide a new method to solve the above technical problems. Summary of the Invention

[0006] In order to achieve the above-mentioned object and other advantages of the present invention, the first object of the present invention is to provide a case library optimization method for digital archive retrieval, comprising the following steps: Preprocess the archival data to be processed to obtain structured data and unstructured data; Discretizing the structured data to obtain discrete features; Based on a global discretized vocabulary, converting the discrete features into structured data feature vectors; Processing the unstructured data in combination with preset archive entity classification information to obtain an unstructured data feature vector; Fusing the structured data feature vector with the unstructured data feature vector to obtain a fused feature; Inputting the fusion features into the case base evaluation model to calculate and predict the probability of retrieval offset based on the current case base; The conversion result of the unstructured data is optimized according to the retrieval offset probability to obtain an optimized instance library.

[0007] Furthermore, the pre-processing of the archival data to be processed includes: The archival data to be processed is cleaned, formatted and normalized.

[0008] Furthermore, the step of discretizing the structured data to obtain discrete features includes: The features of the structured data are converted into discrete features through a tree construction algorithm.

[0009] Furthermore, the step of converting the discrete features into structured data feature vectors based on the global discretized vocabulary includes: Let the discrete features share the global discretization vocabulary; Assign index values to different values of each discrete feature; The index values are mapped into vectors through a learning process and concatenated to obtain a structured data feature vector.

[0010] Furthermore, the step of processing the unstructured data in combination with the preset archive entity classification information to obtain an unstructured data feature vector includes: Get preset archive entity classification information; Performing a word segmentation operation on the unstructured data according to the preset archive entity classification information to obtain a word segmentation result; Using the word segmentation result as input to a pre-trained word vector model to generate a dynamic word vector; A pooling operation is performed on each column in the dynamic word vector to compress the dynamic word vector into a first semantic vector.

[0011] Furthermore, the pre-trained word vector model includes an input layer, an encoder, a prediction layer and an output layer; wherein, The input layer is used to represent the input word segmentation result as a word vector; The encoder includes a multi-layer Transformer encoder, which is used to capture the contextual relationship in the input sequence and encode it into a fixed-length vector representation; each Transformer encoder includes a self-attention layer and a fully connected feedforward layer for feature extraction and representation learning of the input sequence; The prediction layer is used to randomly mask several tokens in the input sequence through a self-attention mechanism, and predict the content of the masked tokens using unmasked words; The output layer is used to perform vector representation on the input sequence to obtain a dynamic word vector.

[0012] Furthermore, the step of processing the unstructured data in combination with the preset archive entity classification information to obtain an unstructured data feature vector further includes: Extracting archival entities from the unstructured data using an archival entity extraction model to obtain an archival entity set; Perform a completion operation on the archive entity set whose number of archive entities does not reach the entity threshold; Inputting each archive entity set into the pre-trained word vector model to obtain a second semantic vector; performing a dimensionality reduction operation on the second semantic vector so that the dimension of the second semantic vector is the same as the dimension of the first semantic vector; The second semantic vector is combined with the first semantic vector to obtain an unstructured data feature vector.

[0013] Furthermore, the step of fusing the structured data feature vector with the unstructured data feature vector to obtain a fused feature includes: The structured data feature vector and the unstructured data feature vector are fused using a probabilistic sparse self-attention mechanism.

[0014] Furthermore, the step of performing feature fusion on the structured data feature vector and the unstructured data feature vector using the probabilistic sparse self-attention mechanism includes: For each input element, for a given query and key, calculate the dot product of the query and the key to get the attention score; The attention scores are screened by a probabilistic method to retain only scores that meet the importance requirements; Normalize the filtered attention scores and calculate the attention weights; The attention weights are used for weighted summation to obtain the final output.

[0015] Furthermore, the case library evaluation model is a neural network model based on a Sigmoid function, and the neural network model adopts a cross entropy loss function; Before inputting the fusion features into the case base evaluation model for prediction, the method further includes: Perform residual and layer normalization operations on the fused features.

[0016] Furthermore, the step of optimizing the conversion result of the unstructured data according to the retrieval offset probability to obtain an optimized case library includes: When the retrieval offset probability reaches a probability threshold, the unstructured data feature vector and the latest conversion result of the unstructured data conversion model are used as inputs of the unstructured data conversion model to obtain a current conversion result; Using the current conversion result as the input of the long short-term memory neural network model at the current timestamp, and combining the hidden state of the long short-term memory neural network model at the previous timestamp and the unit state at the previous moment to perform conversion, thereby obtaining the current conversion result of the unstructured data, as well as the unit state at the current moment and the hidden state at the current moment; The current conversion result of the unstructured data is combined with the structured data to obtain an optimized case library.

[0017] The second object of the present invention is to provide a case library optimization system for digital archive retrieval, which implements the above method and includes an archive data preprocessing module, a structured data discretization processing module, a structured data feature vector generation module, an unstructured data feature vector generation module, a feature fusion module, a retrieval offset prediction module, and a case library optimization module; wherein, The archival data preprocessing module is used to preprocess the archival data to be processed to obtain structured data and unstructured data; The structured data discretization processing module is used to discretize the structured data to obtain discrete features; The structured data feature vector generation module is used to convert the discrete features into structured data feature vectors based on a global discretization vocabulary; The unstructured data feature vector generation module is used to process the unstructured data in combination with the preset archive entity classification information to obtain the unstructured data feature vector; The feature fusion module is used to fuse the structured data feature vector with the unstructured data feature vector to obtain a fused feature; The retrieval offset prediction module is used to input the fusion feature into the case base evaluation model to calculate and predict the retrieval offset probability based on the current case base; The case base optimization module is configured to optimize the conversion result of the unstructured data according to the retrieval offset probability to obtain an optimized case base.

[0018] A third object of the present invention is to provide a computer-readable storage medium having program instructions stored thereon, wherein the program instructions implement the above method when executed.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a case library optimization method, system and storage medium in digital archive retrieval. By using the conversion results of structured data and unstructured data for retrieval offset probability prediction, the retrieval optimization of university digital archives is pushed forward from the retrieval stage to the unstructured data conversion stage. It can discover the data source problem of the retrieval earlier, optimize the conversion results of the unstructured data, reduce the possibility of losing data context and implicit relationships in the structured processing of unstructured data, deeply integrate the external structural features of the data with the internal semantic features to form structured data that can be retrieved, and dynamically adjust the archive retrieval plan to provide archive retrieval with rich semantic information, value information and the association relationship between archive data.

[0020] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and to implement it according to the contents of the description, the following preferred embodiments of the present invention are described in detail with reference to the accompanying drawings. The specific implementation methods of the present invention are given in detail by the following embodiments and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 Flowchart of the case library optimization method in digital archive retrieval of Example 1; Figure 2 A flow chart for generating a structured data feature vector according to Example 1; Figure 3 The unstructured data feature vector generation process of Example 1 Figure 1 ; Figure 4 The unstructured data feature vector generation process of Example 1 Figure 2 ; Figure 5 This is a feature fusion flow chart of Example 1; Figure 6 The conversion result optimization flow chart of the unstructured data in Example 1 is as follows; Figure 7This is a schematic diagram of a case library optimization system in digital archive retrieval according to Example 2; Figure 8 Schematic diagram of the storage medium of Example 3. DETAILED DESCRIPTION

[0022] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. It should be noted that, without conflict, the embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0023] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0024] The figure numbers in this application are only used to distinguish the various steps in the scheme and are not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. Example 1

[0026] A case library optimization method for digital archive retrieval, such as Figure 1 As shown, the following steps are included: S1. Preprocess the archival data to be processed to obtain structured data and unstructured data; Unstructured data in university digital archives refers to data that cannot be stored and queried using traditional database tables, including text, images, audio, and video. Unlike structured data (such as tabular data in a database), unstructured data has no fixed format; its content and structure are typically determined by the data's natural form.

[0027] Since a large amount of data is generated every day in the university archives, including student files, teacher files, school management data, etc., the unstructured big data of university digital archives mainly includes student files, teacher files, school management data, other sources, etc.; among them, student files include students' enrollment registration forms, transcripts, changes in student status, etc., teacher files include teachers' personal resumes, teaching achievements, research achievements, etc., school management data includes school notices, documents, policies, etc., and other sources include library borrowing records, campus one-card, etc. And the data involved in the university archives is of various types, including text, images, audio, video, etc., so these unstructured data have the characteristics of diversity, complexity, large volume, rapid change, etc. The data in the university archives has high value and can be used for student management, teacher evaluation, school decision-making, etc.

[0028] In some embodiments, the preprocessing of the archive data to be processed includes: Cleaning, format conversion, and data normalization processing are performed on the archive data to be processed.

[0029] Unstructured data usually contains a large amount of noise, duplicates, and error information. Through data cleaning, these unnecessary parts can be removed to ensure the accuracy and consistency of the data. The steps of data cleaning include data deduplication, filling missing values, correcting data errors, and standardizing data formats. For example, when processing text data, HTML tags, special characters, and stop words (such as "of", "is", etc.) can be removed. The result of data cleaning is a cleaner and more orderly dataset, laying a foundation for subsequent data conversion and analysis.

[0030] Format conversion is to convert data in different formats (such as text, images, audio) into a unified format for convenient subsequent processing. Data normalization is to scale the data to a specific range to eliminate the dimensional difference between different features.

[0031] S2. Discretize the structured data to obtain discrete features; In some embodiments, the step of discretizing the structured data to obtain discrete features includes: Convert the features of the structured data into discrete features through a tree construction algorithm. For example, the CART algorithm can handle discrete problems and can also handle continuous problems. When dealing with continuous problems, it mainly uses binary splitting to handle continuous variables, that is, if the feature value is greater than a given value, it goes to the left subtree, otherwise it goes to the right subtree. Converting the features of structured data into discrete features reduces the complexity of the data while introducing data semantic information. The converted discrete features can not only express data semantics but also unified the feature form.

[0032] S3. Based on a global discretized vocabulary, convert the discrete features into structured data feature vectors; Further, if Figure 2 As shown, the step of converting the discrete features into structured data feature vectors based on the global discretized vocabulary includes: S31, allowing the discrete features to share the global discretization vocabulary; Among them, the global discretized vocabulary selects features from the data to be processed by using the Apriori algorithm, and uses the leading words and successor words in the mined features to construct the core vocabulary and dependent vocabulary respectively, and filters the candidate sets with less relevance through semantic relevance to obtain the global discretized vocabulary.

[0033] The global discretization vocabulary allows different discrete features to be modeled into the same discrete fine-grained representation.

[0034] S32, assigning an index value to each discrete feature; S33. Map the index values into vectors through a learning process and concatenate them to obtain a structured data feature vector.

[0035] By converting discrete features into semantic vectors through a global discretized vocabulary, not only the semantic information is expanded, but also semantic associations are generated between different discrete features.

[0036] S4. Processing the unstructured data in combination with preset archive entity classification information to obtain an unstructured data feature vector; Optionally, the preset archival entity classification information can be compiled based on the actual archival work of universities and past experience. Specifically, the first-level categories of the entity classification outline are divided into party and group archives, administrative archives, teaching archives, scientific research archives, capital construction archives, instrument and equipment archives, product production and scientific and technological development archives, publishing archives, accounting archives, personal performance archives, and physical archives.

[0037] The secondary categories of the entity classification outline are specifically divided as follows: party and mass, administrative, publishing, and accounting archives are divided into secondary categories according to functional division of labor; teaching archives are divided into secondary categories according to the content, characteristics and formation rules of the document materials; scientific research, infrastructure, instruments and equipment, product production and technological development archives are divided into secondary categories according to projects; personal performance archives of faculty and staff are divided into secondary categories according to the serial number (or code) of their personal department; physical archives are divided into secondary categories according to the type of physical carrier.

[0038] In some embodiments, as Figure 3 As shown, the step of processing the unstructured data in combination with the preset archive entity classification information to obtain the unstructured data feature vector includes: S41. Obtaining preset archive entity classification information; S42, performing a word segmentation operation on the unstructured data according to the preset archive entity classification information to obtain a word segmentation result; Optionally, use a word segmentation tool in Python, such as jieba, to segment the unstructured data according to preset archive entity classification information to obtain a set consisting of multiple entities, conjunctions, punctuation marks, etc.

[0039] S43. Using the word segmentation result as input to a pre-trained word vector model to generate a dynamic word vector; S44. Perform a pooling operation on each column in the dynamic word vector to compress the dynamic word vector into a first semantic vector.

[0040] Pooling integrates the semantic information of dynamic word vectors. For example, average pooling, max pooling, random pooling, and global average pooling can be performed on each column in the dynamic word vector. Pooling can reduce the size of the feature map, which in turn reduces the amount of computation and required video memory. For example, when performing average pooling on each column in the dynamic word vector, only the average of the feature points within the neighborhood is required.

[0041] In some embodiments, the pre-trained word vector model includes an input layer, an encoder, a prediction layer, and an output layer; wherein, The input layer is used to represent the input word segmentation result as a word vector; The encoder includes a multi-layer Transformer encoder, which is used to capture the contextual relationship in the input sequence and encode it into a fixed-length vector representation; each Transformer encoder includes a self-attention layer and a fully connected feedforward layer for feature extraction and representation learning of the input sequence; The prediction layer is used to randomly mask several tokens in the input sequence through a self-attention mechanism, and predict the content of the masked tokens using unmasked words; The output layer is used to perform vector representation on the input sequence to obtain a dynamic word vector.

[0042] In some embodiments, as Figure 4 As shown, the step of processing the unstructured data in combination with the preset archive entity classification information to obtain the unstructured data feature vector further includes: S45. Extracting archival entities from the unstructured data using an archival entity extraction model to obtain an archival entity set; Alternatively, the archival entity extraction model can be trained using a pre-trained natural language processing model using university datasets. The pre-trained natural language processing model can be a BERT model, for example. BERT uses the Transformer architecture, specifically, a multi-layer Transformer encoder.

[0043] S46. Perform a completion operation on the archival entity set whose number of archival entities does not reach the entity threshold; Because the number of entities contained in the archive entity set may be different, in order to unify the subsequent processing, the archive entity set whose number of archive entities does not reach the entity threshold can be supplemented with special characters at the head or tail so that its entity number reaches the entity threshold.

[0044] S47, inputting each archive entity set into the pre-trained word vector model to obtain a second semantic vector; S48: Perform a dimensionality reduction operation on the second semantic vector so that the dimension of the second semantic vector is the same as the dimension of the first semantic vector; The second semantic vector is subjected to dimensionality reduction to facilitate subsequent fusion with the structured data feature vector.

[0045] Optionally, the dimensionality reduction of the second semantic vector is achieved through a fully connected layer. The fully connected layer greatly reduces the dimensionality of the data by flattening the output data of the convolutional layer, thereby reducing the computational burden of the fully connected layer.

[0046] S49: Combine the second semantic vector and the first semantic vector to obtain an unstructured data feature vector.

[0047] S5. Fusing the structured data feature vector with the unstructured data feature vector to obtain a fused feature; In some embodiments, the step of fusing the structured data feature vector with the unstructured data feature vector to obtain a fused feature includes: The structured data feature vector and the unstructured data feature vector are fused using a probabilistic sparse self-attention mechanism.

[0048] Sparse self-attention is an improved self-attention mechanism designed to improve computational efficiency and reduce complexity. In traditional self-attention mechanisms, each input element must be weighted with all other input elements, resulting in high computational complexity. This complexity can become prohibitively high for very long sequences, making it difficult to process long sequences.

[0049] Sparse self-attention introduces a sparse matrix so that each input element only calculates the attention weight with part of the input elements, thereby reducing the computational complexity.

[0050] Specifically, if Figure 5 As shown, the step of performing feature fusion on the structured data feature vector and the unstructured data feature vector using the probabilistic sparse self-attention mechanism includes: S51. For each input element, for a given query and key, calculate the dot product of the query and the key to obtain an attention score; S52. Filter the attention scores using a probabilistic method, and only retain scores that meet the importance requirements; for example, the top k largest scores may be selected, or the scores may be filtered based on a certain threshold.

[0051] S53, normalizing the filtered attention scores and calculating the attention weights; S54: Use the attention weights to perform weighted summation to obtain the final output.

[0052] S6. Inputting the fusion feature into the case base evaluation model to calculate and predict the retrieval offset probability based on the current case base; By predicting and evaluating the retrieval effect of fusion features, the retrieval optimization of university digital archives can be pushed forward from the retrieval stage to the unstructured data conversion stage. This can help discover retrieval data source problems earlier, optimize the conversion results of unstructured data, and reduce the possibility of losing data context and implicit relationships during the structured processing of unstructured data.

[0053] In some embodiments, the case library evaluation model is a neural network model based on a Sigmoid function, and the neural network model adopts a cross entropy loss function; Alternatively, the neural network model can be a feedforward neural network model, in which data passes through different input nodes until it reaches the output node. In contrast to backpropagation, data here only moves in one direction.

[0054] In some embodiments, before inputting the fused features into the case base evaluation model for prediction, the method further includes: The fusion features are subjected to residual and layer normalization operations to solve the gradient disappearance problem.

[0055] S7. Optimize the conversion result of the unstructured data according to the retrieval offset probability to obtain an optimized case library.

[0056] By optimizing the conversion results of unstructured data and deeply integrating the external structural features of the data with the internal semantic features to form structured data that can be retrieved, the role of archival retrieval analysis can be brought into play, and the archival retrieval plan can be dynamically adjusted to provide archival retrieval with rich semantic information, value information and the relationship between archival data.

[0057] In some embodiments, as Figure 6 As shown, the step of optimizing the conversion result of the unstructured data according to the retrieval offset probability to obtain an optimized case library includes: S71. When the retrieval offset probability reaches a probability threshold, the unstructured data feature vector and the latest conversion result of the unstructured data conversion model are used as inputs of the unstructured data conversion model to obtain a current conversion result. Due to the diversity, complexity, volume, and rapid changes of unstructured data, the conversion results of unstructured data can be optimized based on the previous conversion results to achieve efficient conversion of unstructured data. Alternatively, the unstructured data conversion model can adopt the Informer model, which is an optimization of the classic Transformer model.

[0058] S72: Using the current conversion result as the input of the long short-term memory neural network model at the current timestamp, and performing conversion in combination with the hidden state of the long short-term memory neural network model at the previous timestamp and the unit state at the previous moment, to obtain the current conversion result of the unstructured data, as well as the unit state at the current moment and the hidden state at the current moment; The long short-term memory neural network model only needs to process one multidimensional feature at each moment, and the amount of data processed at each moment is greatly reduced, which can reduce the calculation time.

[0059] S73: Combining the current conversion result of the unstructured data with the structured data to obtain an optimized case library.

[0060] This embodiment provides a case library optimization method for digital archive retrieval. By using the conversion results of structured data and unstructured data for retrieval offset probability prediction, the retrieval optimization of digital archives of universities is pushed forward from the retrieval stage to the unstructured data conversion stage. This method can discover the data source problem of the retrieval earlier, optimize the conversion results of the unstructured data, reduce the possibility of losing data context and implicit relationships in the structured processing of unstructured data, deeply integrate the external structural features of the data with the internal semantic features to form structured data that can be retrieved, and dynamically adjust the archive retrieval plan to provide archive retrieval with rich semantic information, value information and the relationship between archive data. Example 2

[0061] Based on the same concept, this embodiment also provides an example library optimization system for digital archive retrieval, which applies the example library optimization method for digital archive retrieval provided in Example 1. For a detailed description of the example library optimization method for digital archive retrieval provided in Example 1, please refer to the corresponding description in Example 1 and will not be repeated here.

[0062] It is understandable that the example library optimization system for digital archive retrieval provided in this embodiment includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in this embodiment, this embodiment can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of this embodiment.

[0063] A case library optimization system for digital archive retrieval 8, such as Figure 7 As shown, it includes an archive data preprocessing module 800, a structured data discretization processing module 810, a structured data feature vector generation module 820, an unstructured data feature vector generation module 830, a feature fusion module 840, a retrieval offset prediction module 850, and an example library optimization module 860; wherein, The archival data preprocessing module is used to preprocess the archival data to be processed to obtain structured data and unstructured data; The structured data discretization processing module is used to discretize the structured data to obtain discrete features; The structured data feature vector generation module is used to convert the discrete features into structured data feature vectors based on a global discretization vocabulary; The unstructured data feature vector generation module is used to process the unstructured data in combination with the preset archive entity classification information to obtain the unstructured data feature vector; The feature fusion module is used to fuse the structured data feature vector with the unstructured data feature vector to obtain a fused feature; The retrieval offset prediction module is used to input the fusion feature into the case base evaluation model to calculate and predict the retrieval offset probability based on the current case base; The case base optimization module is configured to optimize the conversion result of the unstructured data according to the retrieval offset probability to obtain an optimized case base.

[0064] Based on the technical solution of the above embodiment, optionally, the preprocessing of the archival data to be processed includes: The archival data to be processed is cleaned, formatted and normalized.

[0065] Based on the technical solution of the above embodiment, optionally, the step of discretizing the structured data to obtain discrete features includes: The features of the structured data are converted into discrete features through a tree construction algorithm.

[0066] Based on the technical solution of the above embodiment, optionally, the step of converting the discrete features into structured data feature vectors based on the global discretized vocabulary includes: Let the discrete features share the global discretization vocabulary; Assign index values to different values of each discrete feature; The index values are mapped into vectors through a learning process and concatenated to obtain a structured data feature vector.

[0067] Based on the technical solution of the above embodiment, optionally, the step of processing the unstructured data in combination with the preset archive entity classification information to obtain the unstructured data feature vector includes: Get preset archive entity classification information; Performing a word segmentation operation on the unstructured data according to the preset archive entity classification information to obtain a word segmentation result; Using the word segmentation result as input to a pre-trained word vector model to generate a dynamic word vector; A pooling operation is performed on each column in the dynamic word vector to compress the dynamic word vector into a first semantic vector.

[0068] Based on the technical solution of the above embodiment, optionally, the pre-trained word vector model includes an input layer, an encoder, a prediction layer and an output layer; wherein, The input layer is used to represent the input word segmentation result as a word vector; The encoder includes a multi-layer Transformer encoder, which is used to capture the contextual relationship in the input sequence and encode it into a fixed-length vector representation; each Transformer encoder includes a self-attention layer and a fully connected feedforward layer for feature extraction and representation learning of the input sequence; The prediction layer is used to randomly mask several tokens in the input sequence through a self-attention mechanism, and predict the content of the masked tokens using unmasked words; The output layer is used to perform vector representation on the input sequence to obtain a dynamic word vector.

[0069] Based on the technical solution of the above embodiment, optionally, the step of processing the unstructured data in combination with the preset archive entity classification information to obtain the unstructured data feature vector further includes: Extracting archival entities from the unstructured data using an archival entity extraction model to obtain an archival entity set; Perform a completion operation on the archive entity set whose number of archive entities does not reach the entity threshold; Inputting each archive entity set into the pre-trained word vector model to obtain a second semantic vector; performing a dimensionality reduction operation on the second semantic vector so that the dimension of the second semantic vector is the same as the dimension of the first semantic vector; The second semantic vector is combined with the first semantic vector to obtain an unstructured data feature vector.

[0070] Based on the technical solution of the above embodiment, optionally, the step of fusing the structured data feature vector with the unstructured data feature vector to obtain a fused feature includes: The structured data feature vector and the unstructured data feature vector are fused using a probabilistic sparse self-attention mechanism.

[0071] Based on the technical solution of the above embodiment, optionally, the step of performing feature fusion on the structured data feature vector and the unstructured data feature vector using the probabilistic sparse self-attention mechanism includes: For each input element, for a given query and key, calculate the dot product of the query and the key to get the attention score; The attention scores are screened by a probabilistic method to retain only scores that meet the importance requirements; Normalize the filtered attention scores and calculate the attention weights; The attention weights are used for weighted summation to obtain the final output.

[0072] Based on the technical solution of the above embodiment, optionally, the case library evaluation model is a neural network model based on a Sigmoid function, and the neural network model adopts a cross entropy loss function; Before inputting the fusion features into the case base evaluation model for prediction, the method further includes: Perform residual and layer normalization operations on the fused features.

[0073] Based on the technical solution of the above embodiment, optionally, the step of optimizing the conversion result of the unstructured data according to the retrieval offset probability to obtain an optimized case library includes: When the retrieval offset probability reaches a probability threshold, the unstructured data feature vector and the latest conversion result of the unstructured data conversion model are used as inputs of the unstructured data conversion model to obtain a current conversion result; Using the current conversion result as the input of the long short-term memory neural network model at the current timestamp, and combining the hidden state of the long short-term memory neural network model at the previous timestamp and the unit state at the previous moment to perform conversion, thereby obtaining the current conversion result of the unstructured data, as well as the unit state at the current moment and the hidden state at the current moment; The current conversion result of the unstructured data is combined with the structured data to obtain an optimized case library.

[0074] This embodiment provides an instance library optimization system for digital archive retrieval. By using the conversion results of structured data and unstructured data for retrieval offset probability prediction, the retrieval optimization of digital archives of universities is pushed forward from the retrieval stage to the unstructured data conversion stage. This system can discover the data source problems of the retrieval earlier, optimize the conversion results of the unstructured data, reduce the possibility of losing data context and implicit relationships in the structured processing of unstructured data, deeply integrate the external structural features of the data with the internal semantic features to form structured data that can be retrieved, and dynamically adjust the archive retrieval plan to provide archive retrieval with rich semantic information, value information, and the relationship between archive data. Example 3

[0075] A computer-readable storage medium such as Figure 8 As shown, program instructions are stored thereon, and when the program instructions are executed, a case library optimization method for digital archive retrieval is implemented. For a detailed description of the method, reference can be made to the corresponding description in the above method embodiment, which will not be repeated here.

[0076] The number of devices and processing scales described herein are intended to simplify the description of the present invention. Applications, modifications, and variations of the present invention will be readily apparent to those skilled in the art.

[0077] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

[0078] The apparatus, computer device, non-volatile computer storage medium, and method provided in the embodiments of this specification correspond to each other. Therefore, the apparatus, computer device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, computer device, and non-volatile computer storage medium will not be repeated here.

[0079] Those skilled in the art will also appreciate that, in addition to implementing the controller in pure computer-readable program code, it is entirely possible to implement the same functionality by programming the method steps logically, such as through logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered structures within the hardware component. Alternatively, the devices for implementing various functions can be considered both software units implementing the method and structures within the hardware component.

[0080] The systems, devices, or units described in the above embodiments can be implemented by computer chips or physical devices, or by products with certain functions. For ease of description, the above devices are described separately by function, with each unit described separately. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware components.

[0081] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, the embodiments of this specification may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware. Furthermore, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0085] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0086] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program units. Generally, program units include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program units may be located in local and remote computer storage media, including storage devices.

[0087] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0088] The foregoing is merely an example of the present invention and is not intended to limit the present invention to one or more embodiments. It will be apparent to those skilled in the art that various modifications and variations may be made to the present invention to one or more embodiments. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention to one or more embodiments shall be included within the scope of the claims of the present invention to one or more embodiments.

Claims

1. A case library optimization method for digital archive retrieval, characterized in that: The following steps are involved: Preprocess the archival data to be processed to obtain structured data and unstructured data; Discretizing the structured data to obtain discrete features; Based on a global discretized vocabulary, converting the discrete features into structured data feature vectors; Processing the unstructured data in combination with preset archive entity classification information to obtain an unstructured data feature vector; Fusing the structured data feature vector with the unstructured data feature vector to obtain a fused feature; Inputting the fusion features into the case base evaluation model to calculate and predict the probability of retrieval offset based on the current case base; The conversion result of the unstructured data is optimized according to the retrieval offset probability to obtain an optimized instance library.

2. The case library optimization method in digital archive retrieval according to claim 1, characterized in that: The pre-processing of the archive data to be processed includes: The archival data to be processed is cleaned, formatted and normalized.

3. The case library optimization method in digital archive retrieval according to claim 1, characterized in that: The step of discretizing the structured data to obtain discrete features includes: The features of the structured data are converted into discrete features through a tree construction algorithm.

4. The case library optimization method in digital archive retrieval according to claim 3, characterized in that: The step of converting the discrete features into structured data feature vectors based on the global discretized vocabulary includes: Let the discrete features share the global discretization vocabulary; Assign index values to different values of each discrete feature; The index values are mapped into vectors through a learning process and concatenated to obtain a structured data feature vector.

5. The case library optimization method in digital archive retrieval according to claim 1, characterized in that: The step of processing the unstructured data in combination with the preset archive entity classification information to obtain an unstructured data feature vector includes: Get preset archive entity classification information; Performing a word segmentation operation on the unstructured data according to the preset archive entity classification information to obtain a word segmentation result; Using the word segmentation result as input to a pre-trained word vector model to generate a dynamic word vector; A pooling operation is performed on each column in the dynamic word vector to compress the dynamic word vector into a first semantic vector.

6. The case library optimization method for digital archive retrieval according to claim 5, characterized in that: The pre-trained word vector model includes an input layer, an encoder, a prediction layer, and an output layer; wherein, The input layer is used to represent the input word segmentation result as a word vector; The encoder includes a multi-layer Transformer encoder, which is used to capture the contextual relationship in the input sequence and encode it into a fixed-length vector representation; each Transformer encoder includes a self-attention layer and a fully connected feedforward layer for feature extraction and representation learning of the input sequence; The prediction layer is used to randomly mask several tokens in the input sequence through a self-attention mechanism, and predict the content of the masked tokens using unmasked words; The output layer is used to perform vector representation on the input sequence to obtain a dynamic word vector.

7. The case library optimization method in digital archive retrieval according to claim 6, characterized in that: The step of processing the unstructured data in combination with the preset archive entity classification information to obtain an unstructured data feature vector further includes: Extracting archival entities from the unstructured data using an archival entity extraction model to obtain an archival entity set; Perform a completion operation on the archive entity set whose number of archive entities does not reach the entity threshold; Inputting each archive entity set into the pre-trained word vector model to obtain a second semantic vector; performing a dimensionality reduction operation on the second semantic vector so that the dimension of the second semantic vector is the same as the dimension of the first semantic vector; The second semantic vector is combined with the first semantic vector to obtain an unstructured data feature vector.

8. The case library optimization method in digital archive retrieval according to claim 1, characterized in that: The step of fusing the structured data feature vector with the unstructured data feature vector to obtain a fusion feature includes: The structured data feature vector and the unstructured data feature vector are fused using a probabilistic sparse self-attention mechanism.

9. The case library optimization method in digital archive retrieval according to claim 8, characterized in that: The step of performing feature fusion on the structured data feature vector and the unstructured data feature vector using a probabilistic sparse self-attention mechanism includes: For each input element, for a given query and key, calculate the dot product of the query and the key to get the attention score; The attention scores are screened by a probabilistic method to retain only scores that meet the importance requirements; Normalize the filtered attention scores and calculate the attention weights; The attention weights are used for weighted summation to obtain the final output.

10. The case library optimization method for digital archive retrieval according to claim 8, characterized in that: The example library evaluation model is a neural network model based on the Sigmoid function, and the neural network model adopts a cross entropy loss function; Before inputting the fusion features into the case base evaluation model for prediction, the method further includes: Perform residual and layer normalization operations on the fused features.

11. The case library optimization method in digital archive retrieval according to claim 7, characterized in that: The step of optimizing the conversion result of the unstructured data according to the retrieval offset probability to obtain an optimized case library includes: When the retrieval offset probability reaches a probability threshold, the unstructured data feature vector and the latest conversion result of the unstructured data conversion model are used as inputs of the unstructured data conversion model to obtain a current conversion result; Using the current conversion result as the input of the long short-term memory neural network model at the current timestamp, and combining the hidden state of the long short-term memory neural network model at the previous timestamp and the unit state at the previous moment to perform conversion, thereby obtaining the current conversion result of the unstructured data, as well as the unit state at the current moment and the hidden state at the current moment; The current conversion result of the unstructured data is combined with the structured data to obtain an optimized case library.

12. A case library optimization system for digital archive retrieval, implementing the method according to any one of claims 1 to 11, characterized in that: It includes archive data preprocessing module, structured data discretization processing module, structured data feature vector generation module, unstructured data feature vector generation module, feature fusion module, retrieval offset prediction module, and case library optimization module; among them, The archival data preprocessing module is used to preprocess the archival data to be processed to obtain structured data and unstructured data; The structured data discretization processing module is used to discretize the structured data to obtain discrete features; The structured data feature vector generation module is used to convert the discrete features into structured data feature vectors based on a global discretization vocabulary; The unstructured data feature vector generation module is used to process the unstructured data in combination with the preset archive entity classification information to obtain the unstructured data feature vector; The feature fusion module is used to fuse the structured data feature vector with the unstructured data feature vector to obtain a fused feature; The retrieval offset prediction module is used to input the fusion feature into the case base evaluation model to calculate and predict the retrieval offset probability based on the current case base; The case base optimization module is configured to optimize the conversion result of the unstructured data according to the retrieval offset probability to obtain an optimized case base.

13. A computer-readable storage medium, characterized in that Program instructions are stored thereon, and when the program instructions are executed, the method according to any one of claims 1 to 11 is implemented.

Images