A file management method based on artificial intelligence

Abstract

The present invention relates to an archive management method based on artificial intelligence, comprising the following steps: collecting multi-source archive data and preprocessing the multi-source archive data; classifying and storing the preprocessed multi-source archive data through a deep learning model; retrieving the archive data through deep semantic matching when querying the archive data; the present invention accurately classifies the archive data based on the deep learning model, can automatically complete the classification task of a large number of archives, and significantly reduce manpower input; the present invention uses a clustering algorithm to dynamically identify new categories of archives, and can expand the category system without human intervention.

Description

Technical Field

[0001] The invention relates to an archive management method based on artificial intelligence, and belongs to the technical field of data management. Background Art

[0002] As an important tool for information management, archive management system has been widely used in government affairs, enterprises, scientific research and education. With the rapid growth of data volume, traditional archive management system faces more and more challenges.

[0003] Traditional retrieval technology uses keyword-based retrieval methods, such as inverted index and Boolean logic retrieval. After the user enters the keyword, the system returns the results based on the degree of keyword matching. Keyword-based retrieval is prone to miss files that are semantically related but expressed differently.

[0004] Traditional systems are not able to adapt to new user demands (such as new categories and new data types) and require a lot of manual intervention. Summary of the invention

[0005] In order to solve the above problems in the prior art, the present invention is proposed.

[0006] The technical solution of the present invention is as follows:

[0007] In one aspect, the present invention provides an artificial intelligence-based archive management method, comprising the following steps:

[0008] Collect multi-source archival data and pre-process the multi-source archival data;

[0009] Classify and store pre-processed multi-source archival data through deep learning models;

[0010] When querying archival data, archival data is retrieved through deep semantic matching.

[0011] As a preferred embodiment of the present invention, the multi-source archive data includes text data, image data, audio data and video data.

[0012] As a preferred embodiment of the present invention, the preprocessing steps of the multi-source archive data are:

[0013] Define noise filtering rules to filter text data, as shown in the following formula:

[0014] T ′ ={w k ∈T|p(w k )>θ}

[0015] p(w k )=TF(w k )·IDF(ek )

[0016] Where: T′ represents the filtered text data; w k represents the kth word in the text data; p represents the importance score function; p represents the importance score function; θ represents the importance score threshold; TF(w k ) indicates w k The frequency of occurrence in text data; IDF(w k ) indicates w k The inverse document frequency of N represents the total number of text data, n k Indicates that it contains w k The amount of text data.

[0017] As a preferred embodiment of the present invention, the specific steps of classifying the preprocessed multi-source archive data by the deep learning model are:

[0018] The feature extraction of multi-source archive data is shown in the following formula:

[0019] Φ i =σ(W f Z i +b f +∈)

[0020] Where: Φ i represents the classification characteristics of the i-th archive data; W f Represents the classification weight matrix; Z i represents the i-th archive data; b f represents the bias vector; ∈ represents the random noise term; σ represents the activation function;

[0021] A category prediction model is constructed based on the classification features of the archival data, as shown below:

[0022] P(C i |Z i )=Softmax(Φ i β c +γ c )

[0023] Where: P(C i |Z i ) represents archive data Z i Belongs to category C i The probability of c represents the category weight vector; γ c represents category bias; Softmax represents the Softmax function.

[0024] As a preferred embodiment of the present invention, for unclassified archival data, new categories are divided based on the K-means clustering method, and the specific steps are:

[0025] Assume there are K initial categories, and define the center point of each category as shown in the following formula:

[0026]

[0027] Where: μ k represents the center point of the kth category; C k represents the set of archives belonging to the kth category;

[0028] Randomly select K initial center points, assign unclassified archive data to the best center point, and update the center point after the assignment is completed;

[0029] Repeat the above steps until the maximum number of iterations is reached, then end the iteration and obtain the classified archive data;

[0030] Calculate the compactness V of the clustering results for each category, as shown in the following formula:

[0031]

[0032] If the compactness V of the clustering result is greater than the preset threshold, the clustering result is split into two categories;

[0033] Finally, each clustering result is regarded as a new classification.

[0034] As a preferred embodiment of the present invention, the specific steps of performing archive data retrieval through deep semantic matching are:

[0035] Map the query to the semantic space as shown below:

[0036] φ q =σ(W q q+b q )

[0037] Where: φ q represents the semantic embedding of query q; W q represents the embedding weight of query q; b q represents the bias of query q;

[0038] Map the archive to the semantic space as shown below:

[0039] φ d =σ(W d Z d +b d )

[0040] Where: φ drepresents the semantic embedding of the d-th file; W d represents the embedding weight of the dth file; b d represents the bias of the dth file;

[0041] Calculate the semantic similarity between the query and the archive, as shown in the following formula:

[0042]

[0043] Where: S(q,d) represents the semantic similarity between query q and profile d; k represents the supplementary score based on the knowledge graph;

[0044] The file with the greatest semantic similarity is taken as the result of this query.

[0045] On the other hand, the present invention also provides an artificial intelligence-based archive management system, including a data collection module, an archive classification module, and an archive query module;

[0046] The data acquisition module is used to collect multi-source archive data and pre-process the multi-source archive data;

[0047] The archive classification module is used to classify and store the pre-processed multi-source archive data through a deep learning model;

[0048] The archive query module is used to retrieve archive data through deep semantic matching.

[0049] On the other hand, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any embodiment of the present invention when executing the program.

[0050] In yet another aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the present invention.

[0051] The present invention has the following beneficial effects:

[0052] 1. The present invention accurately classifies archival data based on a deep learning model, which can automatically complete the classification task of large quantities of archives and significantly reduce manpower input.

[0053] 2. The present invention utilizes clustering algorithms to dynamically identify new categories of archives and can expand the category system without human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] It should be understood that the step numbers used in this document are only for convenience of description and are not intended to limit the order in which the steps are executed.

[0057] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0058] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0059] The term "and / or" means and includes any and all possible combinations of one or more of the associated listed items.

[0060] Embodiment 1:

[0061] See also Figure 1 , an archive management method based on artificial intelligence, comprising the following steps:

[0062] Collect multi-source archival data and pre-process the multi-source archival data;

[0063] Classify and store pre-processed multi-source archival data through deep learning models;

[0064] When querying archival data, archival data is retrieved through deep semantic matching.

[0065] As a preferred implementation of this embodiment, the multi-source archive data includes text data, image data, audio data and video data.

[0066] As a preferred implementation of this embodiment, the preprocessing steps of the multi-source archive data are:

[0067] Define noise filtering rules to filter text data, as shown in the following formula:

[0068] T ′ ={w k∈T|p(w k )>θ}

[0069] p(w k )=TF(w k )·IDF(w k )

[0070] Where: T′ represents the filtered text data; w k represents the kth word in the text data; p represents the importance score function; p represents the importance score function; θ represents the importance score threshold; TF(w k ) indicates w k The frequency of occurrence in text data; IDF(w k ) indicates w k The inverse document frequency of N represents the total number of text data, n k Indicates that it contains w k The amount of text data.

[0071] As a preferred implementation of this embodiment, the specific steps of classifying the pre-processed multi-source archive data through the deep learning model are:

[0072] The feature extraction of multi-source archive data is shown in the following formula:

[0073] Φ i =σ(W f Z i +b f +∈)

[0074] Where: Φ i represents the classification characteristics of the i-th archive data; W f Represents the classification weight matrix; Z i represents the i-th archive data; b f represents the bias vector; ∈ represents the random noise term; σ represents the activation function;

[0075] A category prediction model is constructed based on the classification features of the archival data, as shown below:

[0076] P(C i |Z i )=Softmax(Φ i β c +γ c )

[0077] Where: P(C i |Z i ) represents archive data Z i Belongs to category C i The probability of crepresents the category weight vector; γ c represents category bias; Softmax represents the Softmax function.

[0078] As a preferred implementation of this embodiment, for unclassified archival data, new categories are divided based on the K-means clustering method. The specific steps are:

[0079] Assume there are K initial categories, and define the center point of each category as shown in the following formula:

[0080]

[0081] Where: μ k represents the center point of the kth category; C k represents the set of archives belonging to the kth category;

[0082] Randomly select K initial center points, assign unclassified archive data to the best center point, and update the center point after the assignment is completed;

[0083] Repeat the above steps until the maximum number of iterations is reached, then end the iteration and obtain the classified archive data;

[0084] Calculate the compactness V of the clustering results for each category, as shown in the following formula:

[0085]

[0086] If the compactness V of the clustering result is greater than the preset threshold, the clustering result is split into two categories;

[0087] Finally, each clustering result is regarded as a new classification.

[0088] As a preferred implementation of this embodiment, the specific steps of performing archive data retrieval through deep semantic matching are:

[0089] Map the query to the semantic space as shown below:

[0090] φ q =σ(W q q+b q )

[0091] Where: φ q represents the semantic embedding of query q; W q represents the embedding weight of query q; b q represents the bias of query q;

[0092] Map the archive to the semantic space as shown below:

[0093] φ d =σ(Wd Z d +b d )

[0094] Where: φ d represents the semantic embedding of the d-th file; W d represents the embedding weight of the dth file; b d represents the bias of the dth file;

[0095] Calculate the semantic similarity between the query and the archive, as shown in the following formula:

[0096]

[0097] Where: S(q,d) represents the semantic similarity between query q and profile d; κ represents the supplementary score based on the knowledge graph;

[0098] The file with the greatest semantic similarity is taken as the result of this query.

[0099] Embodiment 2:

[0100] An artificial intelligence-based archive management system, comprising a data collection module, an archive classification module and an archive query module;

[0101] The data acquisition module is used to collect multi-source archive data and pre-process the multi-source archive data;

[0102] The archive classification module is used to classify and store the pre-processed multi-source archive data through a deep learning model;

[0103] The archive query module is used to retrieve archive data through deep semantic matching.

[0104] The system is used to implement the method in Example 1, which will not be described in detail here.

[0105] Embodiment three:

[0106] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any embodiment of the present invention when executing the program.

[0107] Embodiment 4:

[0108] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any embodiment of the present invention is implemented.

[0109] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0110] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0111] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0112] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.

[0113] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An archive management method based on artificial intelligence, characterized in that: The following steps are involved: Collect multi-source archival data and pre-process the multi-source archival data; Classify and store pre-processed multi-source archival data through deep learning models; When querying archival data, archival data is retrieved through deep semantic matching.

2. The artificial intelligence-based archive management method according to claim 1, characterized in that: The multi-source archive data includes text data, image data, audio data and video data.

3. The artificial intelligence-based archive management method according to claim 2, characterized in that: The preprocessing steps of the multi-source archive data are: Define noise filtering rules to filter text data, as shown in the following formula: T′={w k ∈T|p(w k )>θ} p(w k )=TF(w k )·IDF(w k ) Where: T ′ Represents the filtered text data; w k represents the kth word in the text data; p represents the importance score function; p represents the importance score function; θ represents the importance score threshold; TF(w k ) indicates w k The frequency of occurrence in text data; IDF(w k ) indicates w k The inverse document frequency of N represents the total number of text data, n k Indicates that it contains w k The amount of text data.

4. The artificial intelligence-based archive management method according to claim 1, characterized in that: The specific steps of classifying preprocessed multi-source archival data through deep learning models are: Feature extraction of multi-source archive data is performed as shown in the following formula: F i =σ(W f Z i +b f +∈) Where: Φ i represents the classification characteristics of the i-th archive data; W f Represents the classification weight matrix; Z i represents the i-th archive data; b f represents the bias vector; ∈ represents the random noise term; σ represents the activation function; A category prediction model is constructed based on the classification features of the archival data, as shown below: P(C i |Z i )=Softmax(Φ i ·b c +g c ) Where: P(C i |Z i ) represents archive data Z i Belongs to category C i The probability of c represents the category weight vector; γ c represents category bias; Softmax represents the Softmax function.

5. The artificial intelligence-based archive management method according to claim 4, characterized in that: For unclassified archival data, new categories are divided based on the K-means clustering method. The specific steps are as follows: Assume there are K initial categories, and define the center point of each category as shown in the following formula: Where: μ k represents the center point of the kth category; C k represents the set of archives belonging to the kth category; Randomly select K initial center points, assign unclassified archive data to the best center point, and update the center point after the assignment is completed; Repeat the above steps until the maximum number of iterations is reached, then end the iteration and obtain the classified archive data; Calculate the compactness V of the clustering results for each category, as shown in the following formula: If the compactness V of the clustering result is greater than the preset threshold, the clustering result is split into two categories; Finally, each clustering result is regarded as a new classification.

6. The artificial intelligence-based archive management method according to claim 1, characterized in that: The specific steps for archival data retrieval through deep semantic matching are: Map the query to the semantic space as shown below: f q =σ(W q q+b q ) Where: φ q represents the semantic embedding of query q; W q represents the embedding weight of query q; b q represents the bias of query q; Map the archive to the semantic space as shown below: f d =σ(W d Z d +b d ) Where: φ d represents the semantic embedding of the d-th file; W d represents the embedding weight of the dth file; b d represents the bias of the dth file; Calculate the semantic similarity between the query and the archive, as shown in the following formula: Where: S(q,d) represents the semantic similarity between query q and profile d; κ represents the supplementary score based on the knowledge graph; The file with the greatest semantic similarity is taken as the result of this query.

7. An artificial intelligence-based archive management system, characterized in that: It includes data collection module, file classification module and file query module; The data acquisition module is used to collect multi-source archive data and pre-process the multi-source archive data; The archive classification module is used to classify and store the pre-processed multi-source archive data through a deep learning model; The archive query module is used to retrieve archive data through deep semantic matching.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images