File screening method and device, storage medium and electronic equipment

Abstract

The application discloses a file screening method and device, a storage medium and an electronic equipment. It is related to the field of financial technology. The method comprises the following steps: obtaining a plurality of to-be-screened files, extracting a group of theme keywords for each to-be-screened file through a target model to obtain a plurality of groups of theme keywords; calculating the similarity of each two groups of theme keywords in the plurality of groups of theme keywords to obtain a group of similarities; constructing a similarity network of the plurality of to-be-screened files based on the group of similarities; calculating the network centrality evaluation value of each node in the similarity network to obtain a group of network centrality evaluation values, wherein the network centrality evaluation value is used for evaluating the importance of the keywords in the similarity network; arranging the group of network centrality evaluation values in descending order to obtain a target sequence, and determining the to-be-screened files corresponding to the first N network centrality evaluation values in the target sequence as target files. Through the application, the problem of low accuracy of screening files in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of financial technology, and more specifically, to a document screening method, apparatus, storage medium, and electronic device. Background Technology

[0002] Among related technologies, intelligent human-computer interaction (HCI) plays a crucial role in fields such as intelligent customer service, smart homes, and smart hardware. Question-answering systems (QA) form the technological foundation of HCI. The construction of questions and answers in domain-specific QA systems requires substantial support from specialized knowledge, which is difficult to obtain in open-source environments. Therefore, building domain-specific QA systems necessitates understanding user needs and supplementing them with the experience of experts in the relevant fields. Core users for building the QA system are selected based on the professional level of the experts and recommendations from their peers. The knowledge base of the intelligent QA system is expanded by consulting with these users.

[0003] In related technologies, when filtering professionally relevant documents from a large number of files, expert recommendations are primarily used. However, expert recommendations are highly susceptible to human error, and when the question-and-answer system designer is unfamiliar with the field, it is difficult to quantitatively characterize the expert's expertise in that specific area. Therefore, it cannot be guaranteed that the documents selected by experts meet the needs of the question-and-answer system.

[0004] There is currently no effective solution to the problem of low accuracy in file screening in related technologies. Summary of the Invention

[0005] The main objective of this application is to provide a document screening method, apparatus, storage medium, and electronic device to solve the problem of low accuracy in document screening in related technologies.

[0006] To achieve the above objectives, according to one aspect of this application, a document screening method is provided. The method includes: acquiring multiple documents to be screened; extracting a set of topic keywords from each document to be screened using a target model to obtain multiple sets of topic keywords; calculating the similarity between every two sets of topic keywords to obtain a set of similarity scores; constructing a similarity network for the multiple documents to be screened based on the set of similarity scores, wherein each document to be screened in the similarity network corresponds to a node, and two nodes corresponding to two documents to be screened with a similarity greater than a similarity threshold form an edge, with the similarity between the two nodes having an edge serving as the edge weight; calculating the network centrality evaluation value of each node in the similarity network to obtain a set of network centrality evaluation values, wherein the network centrality evaluation values ​​are used to evaluate the importance of keywords in the similarity network; arranging the set of network centrality evaluation values ​​in descending order to obtain a target sequence, and determining the documents to be screened corresponding to the first N network centrality evaluation values ​​in the target sequence as target documents, where N is a positive integer.

[0007] Optionally, extracting a set of topic keywords for each file to be filtered using the target model includes: receiving the user's filtering request and identifying keywords to be removed that have a relevance lower than the relevance threshold to the filtering request; for each file to be filtered, extracting M topics from the file based on the target model and determining the keyword set under each topic, where M is a positive integer; for each keyword set under each topic, removing keywords to be removed from the keyword set to obtain pending keywords; for each topic, obtaining the frequency of each pending keyword in the file to be filtered, obtaining a set of frequencies, and arranging the set of frequencies in descending order to obtain a frequency sequence; determining the pending keywords corresponding to the first P frequencies in the frequency sequence as topic keywords under the topic, where P is a positive integer; and combining the topic keywords under all topics to obtain a set of topic keywords for the file to be filtered.

[0008] Optionally, calculating the network centrality evaluation value of each node in a similar network includes: determining the node information of each node in the similar network, calculating the betweenness centrality evaluation value, degree centrality evaluation value, and eigenvector centrality evaluation value of the node based on the node information, and determining at least one of the betweenness centrality evaluation value, degree centrality evaluation value, and eigenvector centrality evaluation value as the network centrality evaluation value of the node.

[0009] Optionally, calculating the betweenness centrality evaluation value of a node based on node information includes: dividing all nodes in a similar network into multiple groups of nodes, where each group contains a first node and a second node, the first node being different from the second node; for each group of nodes, determining the path between the first node and the second node, and determining the target path that passes through the target node among all paths between the first node and the second node; calculating the path length of each path between the first node and the second node to obtain a set of path lengths, and determining the shortest first path length in the set of path lengths; calculating the path length of each target path between the first node and the second node to obtain a set of path lengths, and determining the shortest second path length in the set of path lengths; calculating the ratio of the second path length to the first path length to obtain the path length ratio of the group of nodes; and calculating the sum of the path length ratios of multiple groups of nodes to obtain the betweenness centrality evaluation value of the target node.

[0010] Optionally, the path length is calculated by: determining the multiple edges contained in the path and determining the weight of each edge; and calculating the sum of the reciprocals of the weights of all edges to obtain the path length.

[0011] Optionally, calculating the degree centrality evaluation value of a node based on node information includes: determining multiple first neighboring edges connected to the target node and other nodes on each first neighboring edge, wherein other nodes are the other nodes on the first neighboring edge besides the target node; calculating the sum of the weights of the multiple first neighboring edges to obtain the first node strength of the target node; determining multiple second neighboring edges of other nodes on each first neighboring edge, calculating the sum of the weights of the multiple second neighboring edges to obtain the second node strength of other nodes; calculating the sum of the second node strengths of all other nodes, and calculating the ratio of the first node strength to the sum to obtain the degree centrality evaluation value of the target node.

[0012] Optionally, calculating the eigenvector centrality evaluation value of a node based on node information includes: determining the adjacency matrix of the target node in a similar network; and determining the eigenvalues ​​of the eigenvectors of the target node in the adjacency matrix as the eigenvector centrality evaluation value of the target node.

[0013] To achieve the above objectives, according to another aspect of this application, a document screening apparatus is provided. The apparatus includes: an acquisition unit for acquiring multiple documents to be screened, extracting a set of topic keywords from each document using a target model to obtain multiple sets of topic keywords; a first calculation unit for calculating the similarity between every two sets of topic keywords to obtain a set of similarities; a construction unit for constructing a similarity network for the multiple documents to be screened based on the set of similarities, wherein each document to be screened corresponds to a node in the similarity network, and two nodes corresponding to two documents to be screened with a similarity greater than a similarity threshold form an edge, with the similarity between the two nodes having an edge serving as the edge weight; a second calculation unit for calculating the network centrality evaluation value of each node in the similarity network to obtain a set of network centrality evaluation values, wherein the network centrality evaluation values ​​are used to evaluate the importance of keywords in the similarity network; and an arrangement unit for arranging the set of network centrality evaluation values ​​in descending order to obtain a target sequence, and determining the documents to be screened corresponding to the first N network centrality evaluation values ​​in the target sequence as target documents, where N is a positive integer.

[0014] This application employs the following steps: obtaining multiple files to be screened; extracting a set of topic keywords from each file using a target model to obtain multiple sets of topic keywords; calculating the similarity between every two sets of topic keywords to obtain a set of similarity scores; constructing a similarity network for the multiple files to be screened based on the set of similarity scores, wherein each file to be screened corresponds to a node in the similarity network, and two nodes corresponding to two files to be screened with a similarity score greater than a similarity threshold form an edge, with the similarity between the two nodes having an edge serving as the edge weight; calculating the network centrality evaluation value of each node in the similarity network to obtain a set of network centrality evaluation values, wherein the network centrality evaluation values ​​are used to evaluate the importance of keywords in the similarity network; arranging the set of network centrality evaluation values ​​in descending order to obtain a target sequence, and determining the files to be screened corresponding to the first N network centrality evaluation values ​​in the target sequence as target files, where N is a positive integer, thus solving the problem of low accuracy in file screening in related technologies. The target model extracts the topic keywords of each file to be screened, and a similarity network of all files to be screened is constructed based on the similarity between the topic keywords of different files to be screened. The target files to be screened are determined by the network centrality evaluation value of the corresponding node of each file to be screened in the similarity network, thereby improving the accuracy of file screening. Attached Figure Description

[0015] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0016] Figure 1 This is a flowchart of a document filtering method provided according to an embodiment of this application;

[0017] Figure 2 This is a flowchart of a method for extracting subject keywords according to an embodiment of this application;

[0018] Figure 3 This is a schematic diagram of a document screening device provided according to an embodiment of this application;

[0019] Figure 4 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0024] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0025] Network centrality: It represents the importance of each node in a network. Different centralities have different definitions of importance. For example, centrality can be the number of edges that originate from this node, the number of edges that pass through this node, or the number of edges that reach this node.

[0026] LDA: Latent Dirichlet Allocation, which can give the topic of each document in the form of a probability distribution.

[0027] TFIDF: Term Frequency-inverse Document Frequency, can be used to assess the importance of a word in a text.

[0028] The present invention will now be described in conjunction with preferred implementation steps. Figure 1 This is a flowchart of a document filtering method provided according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0029] Step S101: Obtain multiple files to be filtered, and extract a set of topic keywords for each file to be filtered using the target model to obtain multiple sets of topic keywords.

[0030] Specifically, the documents to be screened can be employees' work documents within the company, and the target model can be an LDA model. The work documents contain tags that reflect the employees' professional fields. If different employees have similar professional fields, the tags in their work documents will also be similar. The main content of the work documents includes the job responsibilities that employees need to fulfill and the work achievements they have completed. All of the above content reflects the type of professional field the employees work in. If different employees have similar professional fields, then the work topics and content in the work documents will overlap, meaning that the same words and themes will appear repeatedly in the work documents. Therefore, the tags obtained by extracting themes from the documents to be screened through the target model will also be similar. By extracting a set of theme keywords from each document to be screened using the target model, multiple sets of theme keywords are obtained. Then, by comparing the theme keywords, it can be determined whether there is a correlation between different documents to be screened.

[0031] For example, the files to be screened include employees' work files from the past three years. Employees who joined this year (i.e., those who have never submitted work files) are excluded, and two or more work files from each employee are merged. The LDA model is used to extract a set of operations and maintenance domain tags, or topic keywords, from the files to be screened, such as "MySQL," "switch," "host," and "firewall."

[0032] It's important to note that LDA (Local Theory of Distributed Analysis) is a commonly used topic model for discovering hidden topic structures in text data. LDA assumes that each document is composed of a mixture of multiple topics, and each topic consists of multiple words. The goal of LDA is to infer the topic distribution of each document and the word distribution under each topic based on observed word occurrences. The basic idea of ​​LDA is to represent each document as a probability distribution of topics and each topic as a probability distribution of words. Specifically, for each document, the LDA model assigns a topic to each word and randomly initializes the topic for each word. Then, through multiple iterations, it optimizes the model parameters so that the topic for each word gradually converges to a suitable probability distribution.

[0033] The essence of LDA training is to generate document-topic distribution matrices and topic-word distribution matrices based on the existing dataset. The specific generation process is as follows: For each document to be screened, a topic is extracted from the topic distribution; a word is randomly extracted from the word distribution corresponding to the extracted topic; this process is repeated until every word in the entire document is traversed; the LDA formula is P(word|document) = P(word|topic) × P(topic|document). The LatentDirichletAllocation module from the Python open-source package skleran.decomposition is used to complete the topic word selection and extraction. It is imported using `from sklearn.decomposition import LatentDirichletAllocation`. The topic number parameter `n_components` can be set to 3. Therefore, for each employee, a topic word set containing three topics is extracted from their work documents, and then the 5 words with the highest probability in each topic are selected, ultimately resulting in a set of topic keywords containing 15 topic keywords.

[0034] Step S102: Calculate the similarity between any two sets of topic keywords in multiple sets of topic keywords to obtain a set of similarity scores.

[0035] Specifically, the topic keywords in each group of topic keywords are merged using TF-IDF to generate the word frequency vector of that group of topic keywords. The cosine similarity between the two word frequency vectors corresponding to each pair of topic keywords is calculated as the similarity between the two groups of topic keywords. The larger the similarity value, the more similar the two groups of topic keywords are.

[0036] It should be noted that similarity calculation is implemented using the TfidfVectorizer module in sklearn.feature_extraction.text in Python. The import method is from sklearn.feature_extraction.text import TfidfVectorizer.

[0037] Step S103: Construct a similarity network for multiple files to be screened based on a set of similarities. In the similarity network, each file to be screened corresponds to a node. An edge is formed between two nodes corresponding to two files to be screened with a similarity greater than the similarity threshold. The similarity between two nodes with an edge is used as the weight of the edge.

[0038] Specifically, a similarity network for all files to be screened is constructed by using the similarity between every two sets of files to be screened. In the similarity network, each file to be screened corresponds to a node in the similarity network. An edge is formed between the two nodes corresponding to two files to be screened with a similarity greater than the similarity threshold. The similarity between the two nodes with an edge is used as the weight of the edge.

[0039] It's important to note that social network theory posits that the connections between people constitute a social network. A network consists of nodes and edges connecting them. For example, a friend on your instant messaging app can be considered a node in the network, while the edges connecting those nodes can be seen as friendship relationships within the instant messaging app. In social networks, centrality concepts such as degree centrality, proximity centrality, betweenness centrality, and eigenvector centrality are commonly used to characterize influence.

[0040] Step S104: Calculate the network centrality evaluation value of each node in the similar network to obtain a set of network centrality evaluation values, wherein the network centrality evaluation values ​​are used to evaluate the importance of keywords in the similar network.

[0041] Specifically, network centrality metrics are used to quantitatively characterize the importance of documents to be screened. Network centrality evaluation values ​​can include at least one of the following: betweenness centrality evaluation value, degree centrality evaluation value, and eigenvector centrality evaluation value. The most important nodes in similar networks are determined by calculating the network centrality evaluation value for each node. The calculation of network centrality evaluation values ​​is implemented using Python with the help of the `networkx` module, imported as `import networkx as nx`.

[0042] For example, if we select all three evaluation values—between centrality, degree centrality, and eigenvector centrality—as the network centrality evaluation value, then we can calculate the betweenness centrality, degree centrality, and eigenvector centrality evaluation values ​​for all nodes, resulting in three sets of network centrality evaluation values: one set of betweenness centrality evaluation values, one set of degree centrality evaluation values, and one set of eigenvector centrality evaluation values.

[0043] Step S105: Arrange a set of network centrality evaluation values ​​in descending order to obtain the target sequence, and determine the files to be screened corresponding to the first N network centrality evaluation values ​​in the target sequence as the target files, where N is a positive integer.

[0044] Specifically, the network centrality evaluation values ​​of all nodes are arranged in descending order. A higher network centrality evaluation value indicates that the node is more important in similar networks, meaning that the file corresponding to that node is more important among all the files to be screened. The files corresponding to the top N network centrality evaluation values ​​are extracted from the target sequence as the selected target files.

[0045] For example, select the top 5 files corresponding to the betweenness centrality evaluation values ​​from a set of betweenness centrality evaluation values, select the top 5 files corresponding to the degree centrality evaluation values ​​from a set of degree centrality evaluation values, select the top 5 files corresponding to the eigenvector centrality evaluation values ​​from a set of eigenvector centrality evaluation values, and take the intersection of the 15 files selected above to obtain the target file.

[0046] It should be noted that the top 5 files to be screened are extracted in descending order of network centrality evaluation values ​​as target files, and the core employees to whom these target files belong are identified. These core employees are then consulted to expand the question-and-answer system's knowledge base. By analyzing the working files to extract topic keywords, a similarity network is constructed based on the similarity between these keywords. Then, the network centrality evaluation value is used to screen core employees, and by consulting with these core employees, the goal of expanding the question-and-answer system's knowledge base is achieved. This approach overcomes the personal bias in the selection of core employees and designs a quantitative indicator to characterize professionalism.

[0047] The document filtering method provided in this application involves acquiring multiple documents to be filtered, extracting a set of topic keywords for each document using a target model, resulting in multiple sets of topic keywords; calculating the similarity between any two sets of topic keywords to obtain a set of similarity scores; constructing a similarity network for the multiple documents to be filtered based on this set of similarity scores, where each document to be filtered corresponds to a node in the similarity network, and an edge is formed between the nodes corresponding to two documents to be filtered with a similarity score greater than a similarity threshold, with the similarity between the two nodes having an edge serving as the edge weight; calculating the network centrality evaluation value for each node in the similarity network to obtain a set of network centrality evaluation values, where the network centrality evaluation values ​​are used to evaluate the importance of keywords in the similarity network; arranging the set of network centrality evaluation values ​​in descending order to obtain a target sequence, and determining the documents to be filtered corresponding to the first N network centrality evaluation values ​​in the target sequence as target documents, where N is a positive integer, thus solving the problem of low accuracy in document filtering in related technologies. The target model extracts the topic keywords of each file to be screened, and a similarity network of all files to be screened is constructed based on the similarity between the topic keywords of different files to be screened. The target files to be screened are determined by the network centrality evaluation value of the corresponding node of each file to be screened in the similarity network, thereby improving the accuracy of file screening.

[0048] Thematic keywords are determined based on the frequency of keywords appearing in the documents to be screened. Figure 2 This is a flowchart of a method for extracting subject keywords according to an embodiment of this application, such as... Figure 2 As shown, optionally, in the document filtering method provided in this application embodiment, extracting a set of topic keywords for each document to be filtered through a target model includes: Step S201, receiving the user's filtering request and determining keywords to be removed that have a correlation with the filtering request below a correlation threshold; Step S202, for each document to be filtered, extracting M topics from the document based on the target model and determining a set of keywords under each topic, where M is a positive integer; Step S203, for each set of keywords under a topic, removing keywords to be removed from the set of keywords to obtain undetermined keywords; Step S204, for each topic, obtaining the frequency of each undetermined keyword under the topic in the document to be filtered, obtaining a set of frequencies, and arranging the set of frequencies in descending order to obtain a frequency sequence; Step S205, determining the undetermined keywords corresponding to the first P frequencies in the frequency sequence as topic keywords under the topic, where P is a positive integer; Step S206, combining the topic keywords under all topics to obtain a set of topic keywords for the document to be filtered.

[0049] Specifically, the filtering requirement can be to select keywords related to the operations and maintenance (O&M) professional field. First, a stop word, i.e., the keyword to be removed, is eliminated using an LDA model and the filtering requirements. For example, non-O&M professional field terms such as "effort," "work," and "careful" are removed using an LDA model. Next, an O&M field tag, i.e., themes, is extracted using an LDA model, such as database, equipment, and network domains. Each theme contains a set of keywords under that theme, such as "MySQL" in the database domain, "switch" and "host" in the equipment domain, and "firewall" in the network domain. The LDA model is used to filter the themes and keyword distribution under each theme in the files to be filtered. Then, the keywords are sorted based on their frequency of occurrence under each theme in the files to be filtered, resulting in a frequency sequence. The top P most frequent keywords under each theme are extracted as the theme keywords for that theme. All theme keywords under all themes constitute a set of theme keywords for the files to be filtered. By determining this set of theme keywords for the files to be filtered, the similarity between different files to be filtered can be calculated.

[0050] It's worth noting that the topic can be a tag set in the file to be filtered. Tag recommendations are ubiquitous in daily life. Music websites, Douban, video websites—these entertainment-related websites are replete with tags. Douban's movie tags and book tags, music tags in music software, and video tags on video websites all belong to content-based recommendation algorithms. Similarity-based recommendation algorithms include TF-IDF algorithms, among others.

[0051] Calculating the network centrality evaluation value of each node using node information in similar networks. Optionally, in the document filtering method provided in this application embodiment, calculating the network centrality evaluation value of each node in a similar network includes: determining the node information of each node in the similar network; calculating the betweenness centrality evaluation value, degree centrality evaluation value, and eigenvector centrality evaluation value of the node based on the node information; and determining at least one of the betweenness centrality evaluation value, degree centrality evaluation value, and eigenvector centrality evaluation value as the network centrality evaluation value of the node.

[0052] Specifically, node information can be features related to nodes in similar networks. For example, when calculating the betweenness centrality evaluation value, node information can be the path length from one node to another; when calculating the degree centrality evaluation value, node information can be the node's weight; and when calculating the eigenvector centrality evaluation value, it can be the node's adjacency matrix information. By calculating the network centrality evaluation value of nodes, documents with higher importance can be selected for screening.

[0053] The betweenness centrality evaluation value can be calculated from the path length between two nodes. Optionally, in the document screening method provided in this application embodiment, calculating the betweenness centrality evaluation value of a node based on node information includes: dividing all nodes in a similar network into multiple groups of nodes, wherein each group of nodes includes a first node and a second node, and the first node is different from the second node; for each group of nodes, determining the path between the first node and the second node, and determining the target path that passes through the target node among all paths between the first node and the second node; calculating the path length of each path between the first node and the second node to obtain a set of path lengths, and determining the shortest first path length in the set of path lengths; calculating the path length of each target path between the first node and the second node to obtain a set of path lengths, and determining the shortest second path length in the set of path lengths; calculating the ratio of the second path length to the first path length to obtain the path length ratio of the group of nodes; and calculating the sum of the path length ratios of multiple groups of nodes to obtain the betweenness centrality evaluation value of the target node.

[0054] Specifically, the betweenness centrality evaluation value is determined based on the number of edges passing through a node in a similar network. This is achieved by dividing all nodes in the similar network into multiple groups, determining the paths between each group, and identifying the target path from each group to the target node. Then, the betweenness centrality of the target node is calculated based on the length of the paths between each group and the length of the target path. For example, the betweenness centrality evaluation value of the target node is calculated as follows:

[0055]

[0056] Where WBC(i) is the betweenness centrality evaluation value of node i, i is the target node, s is the first node, t is the second node, and n is the number of nodes in the similar network. Let be the length of the first path from node s to t. Let be the length of the second path from node s to t. This represents the sum of the path length ratios of multiple sets of nodes. The betweenness centrality evaluation value is used to filter out target files with higher importance from multiple files to be screened.

[0057] It should be noted that in similar networks without weights, the betweenness centrality evaluation value of node i is calculated as follows:

[0058]

[0059] Among them, g st Let be the number of all first paths from node s to t. This represents the number of all second paths.

[0060] When calculating the intermediary centrality evaluation value, it is necessary to calculate the path length between two nodes. Optionally, in the document filtering method provided in this application embodiment, the path length is calculated as follows: determine the multiple edges contained in the path and determine the weight of each edge; calculate the sum of the reciprocals of the weights of all edges to obtain the path length.

[0061] Specifically, when the edges of similar networks have weights, the reciprocal of the weights is used to measure the distance. The path length is calculated as follows:

[0062]

[0063] Where nodes m and n are intermediate nodes on the path from node s to node t, w sm w represents the weight of the edge from node s to node m. nt Let be the weight of the edge from node n to node t. The betweenness centrality of the nodes is determined by calculating the path length.

[0064] Calculating the degree centrality evaluation value of a node based on node strength: Optionally, in the document filtering method provided in this application embodiment, calculating the degree centrality evaluation value of a node based on node information includes: determining multiple first neighboring edges connected to the target node and other nodes on each first neighboring edge, wherein other nodes are other nodes on the first neighboring edge besides the target node; calculating the sum of the weights of the multiple first neighboring edges to obtain the first node strength of the target node; determining multiple second neighboring edges of other nodes on each first neighboring edge, calculating the sum of the weights of the multiple second neighboring edges to obtain the second node strength of other nodes; calculating the sum of the second node strengths of all other nodes, and calculating the ratio of the first node strength to the sum to obtain the degree centrality evaluation value of the target node.

[0065] Specifically, the degree centrality evaluation value is determined based on the weights of the neighboring edges of nodes in a similar network. By identifying the neighboring edges of the target node, the first node strength of the target node is calculated based on the weights of these neighboring edges. Other nodes on the neighboring edges are then identified, and their second node strengths are calculated. The degree centrality evaluation value is then calculated based on the first and second node strengths. This value is used to filter files with higher importance. For weighted similar networks, the degree of node i is called its strength, defined as the sum of the weights of all edges connected to node i. The degree centrality evaluation value of the target node is calculated as follows:

[0066]

[0067]

[0068] Among them, w ij Let b be the weight of the edge from node i to node j. iFor the strength of the first node, b j Let WDC(i) be the second node strength of other nodes, and WDC(i) be the degree centrality of the target node i.

[0069] Calculating the eigenvector centrality evaluation value using the adjacency matrix: Optionally, in the document filtering method provided in this application embodiment, calculating the eigenvector centrality evaluation value of a node based on node information includes: determining the adjacency matrix of the target node in a similar network; and determining the eigenvalues ​​of the eigenvectors of the target node in the adjacency matrix as the eigenvector centrality evaluation value of the target node.

[0070] Specifically, the eigenvector centrality evaluation value is determined based on the adjacency matrix of nodes in a similar network. The eigenvalues ​​of the target node's eigenvectors in the adjacency matrix are used as the eigenvector centrality evaluation value. This evaluation value is then used to filter files with higher importance. The calculation method for the eigenvector centrality evaluation value is as follows:

[0071]

[0072] x = cAx;

[0073] Where c is a proportionality constant, after multiple iterations to reach a steady state, it has the matrix form of A, which also means that x is an eigenvalue of A. -1 The corresponding feature vector, WEC(i), is the feature vector centrality evaluation value of the target node i, a ij That is, the eigenvalues ​​of the eigenvectors of the target node i in the adjacency matrix A.

[0074] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0075] This application also provides a file filtering device. It should be noted that the file filtering device of this application can be used to execute the file filtering method provided in this application. The file filtering device provided in this application is described below.

[0076] Figure 3 This is a schematic diagram of a document screening device provided according to an embodiment of this application. For example... Figure 3 As shown, the device includes:

[0077] Unit 10 is used to acquire multiple files to be filtered, and extract a set of topic keywords for each file to be filtered through the target model to obtain multiple sets of topic keywords.

[0078] The first calculation unit 20 is used to calculate the similarity between every two sets of topic keywords in multiple sets of topic keywords, and obtain a set of similarity scores.

[0079] The construction unit 30 is used to construct a similarity network for multiple files to be screened based on a set of similarities. In the similarity network, each file to be screened corresponds to a node. An edge is formed between two nodes corresponding to two files to be screened with a similarity greater than a similarity threshold. The similarity between two nodes with an edge is used as the weight of the edge.

[0080] The second calculation unit 40 is used to calculate the network centrality evaluation value of each node in the similar network, and obtain a set of network centrality evaluation values, wherein the network centrality evaluation values ​​are used to evaluate the importance of keywords in the similar network;

[0081] The permutation unit 50 is used to arrange a set of network centrality evaluation values ​​in descending order to obtain the target sequence, and to determine the files to be screened corresponding to the first N network centrality evaluation values ​​in the target sequence as the target files, where N is a positive integer.

[0082] The document filtering device provided in this application embodiment acquires multiple documents to be filtered through an acquisition unit 10, extracts a set of topic keywords for each document to be filtered through a target model, and obtains multiple sets of topic keywords; a first calculation unit 20 calculates the similarity between every two sets of topic keywords in the multiple sets of topic keywords, and obtains a set of similarity; a construction unit 30 constructs a similarity network for multiple documents to be filtered based on a set of similarity, wherein each document to be filtered in the similarity network corresponds to a node, and two nodes corresponding to two documents to be filtered with a similarity greater than a similarity threshold form an edge, and the similarity between two nodes with an edge is used as the weight of the edge; a second calculation unit 40 calculates the network centrality evaluation value of each node in the similarity network, and obtains a set of network centrality. The evaluation value, where the network centrality evaluation value is used to assess the importance of keywords in the similarity network; the permutation unit 50 arranges a set of network centrality evaluation values ​​in descending order to obtain the target sequence, and determines the files to be screened corresponding to the first N network centrality evaluation values ​​in the target sequence as target files, where N is a positive integer. This solves the problem of low accuracy in screening files in related technologies. The topic keywords of each file to be screened are extracted through the target model, and a similarity network of all files to be screened is constructed based on the similarity between the topic keywords of different files to be screened. The target files are determined by the network centrality evaluation value of the node corresponding to each file to be screened in the similarity network, thereby improving the accuracy of screening files.

[0083] Optionally, in the document filtering device provided in this application embodiment, the acquisition unit 10 includes: a receiving module, used to receive the user's filtering request and determine keywords to be removed whose relevance to the filtering request is lower than the relevance threshold; an extraction module, used to extract M topics from the document to be filtered based on the target model for each document to be filtered, and determine the keyword set under each topic, where M is a positive integer; a removal module, used to remove keywords to be removed from the keyword set under each topic to obtain undetermined keywords; an acquisition module, used to acquire the frequency of each undetermined keyword under each topic in the document to be filtered, obtain a set of frequencies, and arrange the set of frequencies in descending order to obtain a frequency sequence; a first determination module, used to determine the undetermined keywords corresponding to the first P frequencies in the frequency sequence as topic keywords under the topic, where P is a positive integer; and a combination module, used to combine the topic keywords under all topics to obtain a set of topic keywords for the document to be filtered.

[0084] Optionally, in the document filtering device provided in this application embodiment, the second calculation unit 40 includes: a second determining module, used to determine the node information of each node in a similar network, and calculate the betweenness centrality evaluation value, degree centrality evaluation value and eigenvector centrality evaluation value of the node based on the node information; and a third determining module, used to determine at least one of the betweenness centrality evaluation value, degree centrality evaluation value and eigenvector centrality evaluation value as the network centrality evaluation value of the node.

[0085] Optionally, in the document screening device provided in this application embodiment, the second determining module includes: a partitioning submodule, used to partition all nodes in a similar network into multiple groups of nodes, wherein each group of nodes includes a first node and a second node, the first node being different from the second node; a first determining submodule, used to determine, for each group of nodes, the path between the first node and the second node, and to determine the target path that passes through the target node among all paths between the first node and the second node; a first calculation submodule, used to calculate the path length of each path between the first node and the second node, to obtain a set of path lengths, and to determine the shortest first path length in the set of path lengths; a second calculation submodule, used to calculate the path length of each target path between the first node and the second node, to obtain a set of path lengths, and to determine the shortest second path length in the set of path lengths; a third calculation submodule, used to calculate the ratio of the second path length to the first path length, to obtain the path length ratio of the group of nodes; and a fourth calculation submodule, used to calculate the sum of the path length ratios of multiple groups of nodes, to obtain the betweenness centrality evaluation value of the target node.

[0086] Optionally, in the file filtering device provided in this application embodiment, the path length is calculated as follows: determine the multiple edges contained in the path and determine the weight of each edge; calculate the sum of the reciprocals of the weights of all edges to obtain the path length.

[0087] Optionally, in the document screening device provided in this application embodiment, the second determining module includes: a second determining submodule, used to determine multiple first neighboring edges connected to the target node and other nodes on each first neighboring edge, wherein the other nodes are another node on the first neighboring edge besides the target node; a fifth calculation submodule, used to calculate the sum of the weights of the multiple first neighboring edges to obtain the first node strength of the target node; a third determining submodule, used to determine multiple second neighboring edges of other nodes on each first neighboring edge, calculate the sum of the weights of the multiple second neighboring edges to obtain the second node strength of other nodes; and a sixth calculation submodule, used to calculate the sum of the second node strengths of all other nodes, and calculate the ratio of the first node strength to the sum to obtain the degree centrality evaluation value of the target node.

[0088] Optionally, in the document screening device provided in this application embodiment, the second determining module includes: a fourth determining submodule, used to determine the adjacency matrix of the target node in the similar network; and a fifth determining submodule, used to determine the eigenvalues ​​of the eigenvectors of the target node in the adjacency matrix as the eigenvector centrality evaluation value of the target node.

[0089] The document filtering device includes a processor and a memory. The aforementioned acquisition unit 10, first calculation unit 20, construction unit 30, second calculation unit 40, and sorting unit 50 are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to achieve the corresponding functions.

[0090] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can improve the accuracy of file filtering.

[0091] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0092] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements a file filtering method.

[0093] This invention provides a processor for running a program, wherein the program executes a file filtering method during runtime.

[0094] Figure 4This is a schematic diagram of an electronic device provided according to an embodiment of this application. For example... Figure 4 As shown, the electronic device 401 includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: acquiring multiple files to be screened; extracting a set of topic keywords from each file using a target model to obtain multiple sets of topic keywords; calculating the similarity between every two sets of topic keywords to obtain a set of similarity scores; constructing a similarity network for the multiple files to be screened based on the set of similarity scores, where each file to be screened corresponds to a node in the similarity network, and two nodes corresponding to two files with a similarity score greater than a similarity threshold form an edge, with the similarity between the two nodes forming the edge weight; calculating the network centrality evaluation value of each node in the similarity network to obtain a set of network centrality evaluation values, where the network centrality evaluation value is used to evaluate the importance of keywords in the similarity network; arranging the set of network centrality evaluation values ​​in descending order to obtain a target sequence, and determining the files to be screened corresponding to the first N network centrality evaluation values ​​in the target sequence as target files, where N is a positive integer. The device in this paper can be a server, PC, PAD, mobile phone, etc.

[0095] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: acquiring multiple files to be screened; extracting a set of topic keywords from each file to be screened using a target model to obtain multiple sets of topic keywords; calculating the similarity between every two sets of topic keywords to obtain a set of similarity; constructing a similarity network for multiple files to be screened based on the set of similarity, wherein each file to be screened in the similarity network corresponds to a node, and two nodes corresponding to two files to be screened with a similarity greater than a similarity threshold form an edge, and the similarity between two nodes with an edge is used as the weight of the edge; calculating the network centrality evaluation value of each node in the similarity network to obtain a set of network centrality evaluation values, wherein the network centrality evaluation value is used to evaluate the importance of keywords in the similarity network; arranging the set of network centrality evaluation values ​​in descending order to obtain a target sequence, and determining the files to be screened corresponding to the first N network centrality evaluation values ​​in the target sequence as target files, wherein N is a positive integer.

[0096] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0100] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0101] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0102] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0103] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0104] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A document filtering method, characterized in that, include: Multiple files to be filtered are obtained, and a set of topic keywords are extracted from each file using a target model to obtain multiple sets of topic keywords. The target model is used to infer the topic distribution of the files to be filtered and the vocabulary distribution under each topic based on the occurrence of words in the files to be filtered. Calculate the similarity between any two sets of topic keywords in the multiple sets of topic keywords to obtain a set of similarity scores; Based on the set of similarities, a similarity network is constructed for the multiple files to be screened. In the similarity network, each file to be screened corresponds to a node. An edge is formed between two nodes corresponding to two files to be screened with a similarity greater than a similarity threshold. The similarity between two nodes with an edge is used as the weight of the edge. Calculate the network centrality evaluation value of each node in the similar network to obtain a set of network centrality evaluation values, wherein the network centrality evaluation values ​​are used to evaluate the importance of the keyword in the similar network; The set of network centrality evaluation values ​​are arranged in descending order to obtain the target sequence, and the files to be screened corresponding to the first N network centrality evaluation values ​​in the target sequence are determined as target files, where N is a positive integer; Calculating the network centrality evaluation value of each node in the similar network includes: Determine the node information of each node in the similar network, and calculate the betweenness centrality evaluation value, degree centrality evaluation value, and eigenvector centrality evaluation value of the node based on the node information; At least one of the betweenness centrality evaluation value, the degree centrality evaluation value, and the eigenvector centrality evaluation value is determined as the network centrality evaluation value of the node.

2. The method according to claim 1, characterized in that, The target model is used to extract a set of topic keywords for each document to be screened, including: Receive the user's filtering request and determine the keywords to be removed that have a correlation with the filtering request that is lower than the correlation threshold; For each file to be screened, M topics are extracted from the file based on the target model, and a set of keywords under each topic is determined, where M is a positive integer; For each topic's keyword set, remove the keywords to be removed from the keyword set to obtain the keywords to be determined; For each topic, obtain the frequency of each undetermined keyword under the topic in the files to be filtered, obtain a set of frequencies, and arrange the set of frequencies in descending order to obtain a frequency sequence; The undetermined keywords corresponding to the first P frequencies in the frequency sequence are determined as the topic keywords under the topic, where P is a positive integer; Combine the keywords of all the topics to obtain a set of keywords for the documents to be screened.

3. The method according to claim 1, characterized in that, The calculation of the betweenness centrality evaluation value of the node based on the node information includes: All nodes in the similar network are divided into multiple groups of nodes, wherein each group of nodes contains a first node and a second node, and the first node is different from the second node; For each group of nodes, determine the path between the first node and the second node, and determine the target path that passes through the target node among all paths between the first node and the second node; Calculate the path length of each path between the first node and the second node to obtain a set of path lengths, and determine the shortest first path length in the set of path lengths; Calculate the path length of each target path between the first node and the second node to obtain a set of path lengths, and determine the shortest second path length in the set of path lengths; Calculate the ratio of the second path length to the first path length to obtain the path length ratio of the group nodes; The sum of the path length ratios of the multiple sets of nodes is calculated to obtain the betweenness centrality evaluation value of the target node.

4. The method according to claim 3, characterized in that, The path length is calculated in the following way: Identify the multiple edges contained in the path and determine the weight of each edge; The path length is obtained by summing the reciprocals of the weights of all edges.

5. The method according to claim 1, characterized in that, The degree centrality evaluation value of the node is calculated based on the node information, including: Determine multiple first adjacent edges connected to the target node and other nodes on each first adjacent edge, wherein the other nodes are another node on the first adjacent edge besides the target node; The sum of the weights of the plurality of first neighboring edges is calculated to obtain the first node strength of the target node; Determine multiple second neighboring edges of other nodes on each first neighboring edge, calculate the sum of the weights of the multiple second neighboring edges, and obtain the second node strength of the other nodes; The degree centrality evaluation value of the target node is obtained by summing the second node strengths of all other nodes and calculating the ratio of the first node strength to the sum.

6. The method according to claim 1, characterized in that, Calculating the eigenvector centrality evaluation value of the node based on the node information includes: Determine the adjacency matrix of the target node in the similar network; The eigenvalues ​​of the eigenvectors of the target node in the adjacency matrix are determined as the eigenvector centrality evaluation values ​​of the target node.

7. A document screening device, characterized in that, include: The acquisition unit is used to acquire multiple files to be filtered, and extract a set of topic keywords for each file to be filtered through the target model to obtain multiple sets of topic keywords. The target model is used to infer the topic distribution of the files to be filtered and the vocabulary distribution under each topic based on the occurrence of words in the files to be filtered. The first calculation unit is used to calculate the similarity between every two sets of topic keywords in the multiple sets of topic keywords, and obtain a set of similarity scores. The construction unit is used to construct a similarity network for the multiple files to be screened based on the set of similarities. In the similarity network, each file to be screened corresponds to a node. An edge is formed between two nodes corresponding to two files to be screened with a similarity greater than a similarity threshold. The similarity between two nodes with an edge is used as the weight of the edge. The second calculation unit is used to calculate the network centrality evaluation value of each node in the similar network, and obtain a set of network centrality evaluation values, wherein the network centrality evaluation values ​​are used to evaluate the importance of the keyword in the similar network; The permutation unit is used to arrange the set of network centrality evaluation values ​​in descending order to obtain the target sequence, and to determine the files to be screened corresponding to the first N network centrality evaluation values ​​in the target sequence as target files, where N is a positive integer; The second calculation unit includes: a second determining module, used to determine the node information of each node in the similar network, and calculate the betweenness centrality evaluation value, degree centrality evaluation value and eigenvector centrality evaluation value of the node based on the node information; and a third determining module, used to determine at least one of the betweenness centrality evaluation value, degree centrality evaluation value and eigenvector centrality evaluation value as the network centrality evaluation value of the node.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein the program, when running, controls the device where the non-volatile storage medium is located to execute the file filtering method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the file filtering method according to any one of claims 1 to 6.

Images