News search method and device, electronic device, and storage medium

Abstract

The application relates to a news retrieval method and device, an electronic device and a storage medium. The method comprises the following steps: receiving a retrieval request, and extracting a first keyword from the retrieval request; acquiring a news keyword index, a keyword news index, a word vector index, a clustered keyword index and a word cluster vector index; determining a target keyword cluster based on a first word vector corresponding to the first keyword and the word cluster vector index, taking the target keyword cluster as an initial index value, and sequentially indexing the clustered keyword index and the keyword news index to obtain a candidate news text; indexing a second keyword corresponding to the candidate news text in the news keyword index, indexing a second word vector corresponding to the second keyword from the word vector index, and retrieving a target news text from the candidate news text based on the semantic similarity between the first word vector and the second word vector. Through the application, the problem that a semantic retrieval system cannot quickly perform semantic retrieval under limited computing resources is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and in particular to a news retrieval method and device, an electronic device, and a storage medium. BACKGROUND

[0002] In related technologies, document retrieval is mostly based on text matching, and the results are returned through the matching degree of related texts. Using text matching based on text matching, it is difficult to meet the needs of users to query documents or texts based on semantics. When a user queries, the user needs to know exactly what content they want to query, for example, when searching for "basketball", unless the document contains "basketball", the query result cannot be obtained.

[0003] In related technologies, semantic retrieval systems use algorithm-based matching to compensate for the semantic-level defects of text matching. However, semantic retrieval based on algorithm matching requires a very large amount of computing resources, and it is difficult to expand on the order of document size, so it is impossible to achieve fast semantic retrieval under the condition of limited computing resources.

[0004] There is no better technical solution to the problem that the semantic retrieval system in related technologies cannot achieve fast semantic retrieval under the condition of limited computing resources. SUMMARY

[0005] The present application provides a news retrieval method, device, electronic device, and storage medium to at least solve the problem that the semantic retrieval system in related technologies cannot achieve fast semantic retrieval under the condition of limited computing resources.

[0006] In a first aspect, the present application provides a news retrieval method, comprising: receiving a retrieval request of a user, and extracting a first keyword from the retrieval request, wherein the first keyword is used to represent a target news text that the user needs to retrieve; obtaining a constructed hierarchical index dataset, wherein the hierarchical index dataset includes a news keyword index, a keyword news index, a word vector index, a clustered keyword index, and a word cluster vector index; based on a first word vector corresponding to the first keyword and the word cluster vector index, determining a target keyword cluster, and taking the target keyword cluster as an initial index value, sequentially indexing in the clustered keyword index and the keyword news index, to obtain a plurality of candidate news texts; indexing a second keyword corresponding to the candidate news text in the news keyword index, and after indexing a second word vector corresponding to the second keyword from the word vector index, based on the semantic similarity of the first word vector and the second word vector, retrieving the target news text from the plurality of candidate news texts.

[0007] In a second aspect, the present application provides a news retrieval device, comprising:

[0008] receiving a search request of a user, and extracting a first keyword from the search request, wherein the first keyword is used to represent a target news text of the user demand search;

[0009] obtaining a constructed hierarchical index dataset, wherein the hierarchical index dataset comprises a news keyword index, a keyword news index, a word vector index, a clustered keyword index and a word cluster vector index;

[0010] preliminary screening, based on a first word vector corresponding to the first keyword and the word cluster vector index, determining a target keyword cluster, and taking the target keyword cluster as an initial index value, sequentially indexing the clustered keyword index and the keyword news index to obtain a plurality of candidate news texts;

[0011] processing, in the news keyword index, indexing a second keyword corresponding to the candidate news text, and after indexing a second word vector corresponding to the second keyword from the word vector index, based on semantic similarity of the first word vector and the second word vector, searching the target news text from the plurality of candidate news texts.

[0012] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

[0013] The memory is used to store a computer program.

[0014] The processor is used to execute the program stored on the memory, and implement the steps of the news search method according to any one of the embodiments of the first aspect.

[0015] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps of the news search method according to any one of the embodiments of the first aspect.

[0016] Compared with the related art, the news retrieval method, device, electronic device and storage medium provided in the embodiment can receive a retrieval request of a user, and extract a first keyword from the retrieval request, wherein the first keyword is used to represent a target news text required by the user; a constructed hierarchical index dataset is acquired, wherein the hierarchical index dataset includes a news keyword index, a keyword news index, a word vector index, a clustered keyword index and a word cluster vector index; based on a first word vector corresponding to the first keyword and the word cluster vector index, a target keyword cluster is determined, and the target keyword cluster is used as an initial index value to sequentially index the clustered keyword index and the keyword news index to obtain a plurality of candidate news texts; in the news keyword index, a second keyword corresponding to the candidate news text is indexed, and after a second word vector corresponding to the second keyword is indexed from the word vector index, based on semantic similarity of the first word vector and the second word vector, the target news text is retrieved from the plurality of candidate news texts, thereby solving the problem that the semantic retrieval system in the related art cannot realize fast semantic retrieval under the condition of limited computing resources, and achieving the beneficial effects of fast semantic retrieval under low resource conditions, semantic matching of the semantic retrieval system and speed balance.

[0017] The details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the description.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required to be used in the embodiments or prior art description will be briefly introduced as follows, and obviously, other drawings can also be obtained by those skilled in the art without creative labor under the premise of these drawings.

[0020] Figure 1 A flowchart of a news retrieval method provided in the embodiment of the present application;

[0021] Figure 2 is a schematic diagram of a semantic retrieval system corresponding to the preferred embodiment of the present application;

[0022] Figure 3 is a schematic diagram of the formation of the hierarchical index dataset in the preferred embodiment of the present application;

[0023] Figure 4A flowchart of a news retrieval method according to an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the news retrieval method according to an embodiment of the present application includes the following steps:

[0024] Figure 5 A flowchart of a news retrieval method according to an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the news retrieval method according to an embodiment of the present application includes the following steps:

[0025] Figure 6 A flowchart of a news retrieval method according to an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the news retrieval method according to an embodiment of the present application includes the following steps:

[0026] Figure 7 A flowchart of a news retrieval method according to an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the news retrieval method according to an embodiment of the present application includes the following steps:

[0027] Figure 8 A flowchart of a news retrieval method according to an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the news retrieval method according to an embodiment of the present application includes the following steps: DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0029] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0030] Figure 1 A flowchart of a news retrieval method according to an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the news retrieval method according to an embodiment of the present application includes the following steps: Figure 1

[0031] Step S101, receiving a retrieval request of a user and extracting a first keyword from the retrieval request, wherein the first keyword is used to represent a target news text required by the user for retrieval.

[0032] In the present embodiment, the execution subject of the news retrieval method according to an embodiment of the present application is a semantic retrieval system running on a terminal, a server or a cloud. When the semantic retrieval system receives a retrieval request of a user, that is, a user query, the semantic retrieval system will adopt natural language processing (NLP) to extract keywords from the retrieval request, so as to obtain a first keyword used to represent a target news text required by the user for retrieval, that is, the target news text at least includes the first keyword, for example, the Textrank algorithm is adopted to extract keywords.

[0033] ​In the embodiment, the first keyword is not a single keyword, and can be an entity such as a person name, an organization name, or a story location appearing in the target news text, or a keyword for summarizing, summarizing, or evaluating the target news text, for example, a rare landscape or natural comfort. Therefore, in the embodiment, the extracted first keyword is presented in a keyword list.

[0034] In step S102, the constructed hierarchical index dataset is obtained, wherein the hierarchical index dataset includes a news keyword index, a keyword news index, a word vector index, a clustered keyword index, and a word cluster vector index.

[0035] In the embodiment, the constructed index is used to quickly complete the retrieval. Therefore, after receiving the query request of the user and extracting the first keyword, the news semantic retrieval is implemented based on the first keyword and the hierarchical index dataset including multiple indexes. Before the retrieval, the constructed hierarchical index dataset is obtained from the corresponding index database.

[0036] In step S103, based on the first word vector corresponding to the first keyword and the word cluster vector index, the target keyword cluster is determined, and the target keyword cluster is used as an initial index value to sequentially index the clustered keyword index and the keyword news index to obtain multiple candidate news texts.

[0037] In the embodiment, after the first keyword is extracted, the first keyword is vectorized in the same way as the word vector in the generated word vector index. Specifically, the keyword vectorization method includes but is not limited to one of the following methods: using a pre-trained language model, using vector training, and calling an open source vector. The first keyword is processed by using any of the above keyword vectorization methods to obtain the corresponding first word vector.

[0038] In the embodiment, after the first word vector corresponding to the first keyword is determined, the semantic distance is calculated with the word cluster vector index, that is, the cosine distance is calculated based on the first word vector and the corresponding word cluster vector in the word cluster vector index, and the first K clusters with the smallest distance are taken to determine the target keyword cluster. Then, the corresponding keywords are indexed according to the target keyword index, and the corresponding news texts are indexed again according to the indexed keywords to obtain the preliminary screening news texts, that is, the candidate news texts. Of course, in the embodiment, the candidate news texts are not single news texts, but multiple news texts.

[0039] In step S104, the second keyword corresponding to the candidate news text is indexed in the news keyword index, and after the second word vector corresponding to the second keyword is indexed from the word vector index, the target news text is retrieved from the multiple candidate news texts based on the semantic similarity of the first word vector and the second word vector.

[0040] In the embodiment, after the candidate news texts are screened out, the news semantic retrieval is performed on the candidate news texts, that is, the news keywords corresponding to the candidate news texts, that is, the second keywords, are indexed first, then the second keywords are indexed by the word vector, to obtain the second word vector corresponding to the second keywords, and then the semantic distance is calculated between the first word vector and the second word vector, so as to obtain the semantic matching degree of each candidate news text corresponding to the query request, so as to screen out the target news text matching the query request of the user.

[0041] Through the steps S101 to S104, the retrieval request of the user is received, and the first keyword is extracted from the retrieval request, wherein the first keyword is used to represent the target news text required by the user to retrieve; the constructed hierarchical index dataset is obtained, wherein the hierarchical index dataset includes a news keyword index, a keyword news index, a word vector index, a clustering keyword index, and a word clustering vector index; based on the first word vector corresponding to the first keyword and the word clustering vector index, the target keyword cluster is determined, and the target keyword cluster is used as the initial index value to sequentially index the clustering keyword index and the keyword news index, to obtain a plurality of candidate news texts; in the news keyword index, the second keyword corresponding to the candidate news text is indexed, and after the second keyword corresponding to the second word vector is indexed from the word vector index, the target news text is retrieved from the plurality of candidate news texts based on the semantic similarity between the first word vector and the second word vector, thereby solving the problem that the semantic retrieval system in the related art cannot realize fast semantic retrieval under the condition of limited computing resources, and achieving the beneficial effects of fast semantic retrieval under low resource conditions, semantic matching of the semantic retrieval system, and balance of speed.

[0042] In some embodiments, based on the first word vector corresponding to the first keyword and the word clustering vector index, the target keyword cluster is determined by the following steps:

[0043] Step 21, respectively acquiring a word vector matrix composed of all first word vectors and a word clustering vector matrix composed of all word clustering vectors in the word clustering vector index.

[0044] Step 22, performing matrix point multiplication operation based on class cosine similarity on the word vector matrix and the word clustering vector matrix to generate a keyword cluster set, wherein the keyword cluster set includes keyword clusters and cluster weights corresponding to the keyword clusters.

[0045] Step 23, based on the cluster weights and a preset rule, selecting a preset number of keyword clusters from the keyword cluster set to obtain the target keyword cluster.

[0046] The word vector matrix composed of all the first word vectors and the word cluster vector matrix composed of all the word cluster vectors in the word cluster vector index are obtained respectively through the above steps; the matrix dot product operation based on the class cosine similarity is performed on the word vector matrix and the word cluster vector matrix to generate a keyword cluster set, wherein the keyword cluster set includes keyword clusters and cluster weights corresponding to the keyword clusters; based on the cluster weights and the preset rule, a preset number of keyword clusters are selected from the keyword cluster set to obtain target keyword clusters; the class cosine similarity is used to quickly obtain cluster sorting and realize fast retrieval of clusters.

[0047] In some embodiments, based on the cluster weights and the preset rule, a preset number of keyword clusters are selected from the keyword cluster set to obtain target keyword clusters, including the following steps: selecting a preset number of keyword clusters from the keyword cluster set in descending order of the cluster weights to obtain the target keyword clusters.

[0048] In some embodiments, the target keyword clusters are used as initial index values to sequentially perform indexing in the cluster keyword index and the keyword news index to obtain a plurality of candidate news texts, which are achieved by the following steps:

[0049] Step 31, obtaining a cluster keyword index and a keyword news index, wherein the cluster keyword index includes clusters, index keywords, and relationship information between the clusters and the index keywords, and the keyword news index includes index keywords, first news texts, and relationship information between the index keywords and the first news texts.

[0050] Step 32, in the cluster keyword index, querying the index keywords corresponding to the target keyword clusters, and in the keyword news index, querying the first news texts corresponding to the index keywords to obtain a plurality of candidate news texts.

[0051] In the above steps, the cluster keyword index and the keyword news index are obtained, wherein the cluster keyword index includes clusters, index keywords, and relationship information between the clusters and the index keywords, and the keyword news index includes index keywords, first news texts, and relationship information between the index keywords and the first news texts; in the cluster keyword index, the index keywords corresponding to the target keyword clusters are queried, and in the keyword news index, the first news texts corresponding to the index keywords are queried to obtain a plurality of candidate news texts, which realizes fast retrieval of news keywords and initial screening of news texts based on query indexing, and a large number of news can be pruned by retrieving clusters to obtain keywords and news.

[0052] In some embodiments, in the news keyword index, the second keywords corresponding to the candidate news texts are indexed, including the following steps:

[0053] In step 41, a news keyword index is obtained, wherein the news keyword index comprises second news texts, news keywords, and relationship information between the second news texts and the news keywords.

[0054] In step 42, the candidate news text is taken as a second news text, the corresponding news keyword is queried in the news keyword index, and the news keyword is taken as a second keyword corresponding to the candidate news text.

[0055] By obtaining the news keyword index in the above steps, wherein the news keyword index comprises second news texts, news keywords, and relationship information between the second news texts and the news keywords; taking the candidate news text as a second news text, querying the corresponding news keyword in the news keyword index, and taking the news keyword as a second keyword corresponding to the candidate news text, the corresponding index is constructed based on the news keyword index, and the corresponding keyword of the news is quickly retrieved.

[0056] In some embodiments, indexing the second keyword corresponding second word vector from the word vector index comprises the following steps:

[0057] In step 51, a word vector index is obtained, wherein the word vector index comprises candidate keywords, keyword vectors, and relationship information between the candidate keywords and the keyword vectors.

[0058] In step 52, the second keyword is taken as a candidate keyword, the corresponding keyword vector is queried in the word vector index, and the keyword vector is taken as a second keyword corresponding second word vector.

[0059] By obtaining the word vector index in the above steps, wherein the word vector index comprises candidate keywords, keyword vectors, and relationship information between the candidate keywords and the keyword vectors; taking the second keyword as a candidate keyword, querying the corresponding keyword vector in the word vector index, and taking the keyword vector as a second keyword corresponding second word vector, the keyword vector corresponding to the keyword of the news is quickly determined based on the corresponding index that has been constructed, and data is quickly provided for subsequent calculation of the semantic distance to determine the document matching degree.

[0060] In some embodiments, based on the semantic similarity of the first word vector and the second word vector, a target news text is retrieved from a plurality of candidate news texts, comprising the following steps:

[0061] In step 61, a first vector matrix composed of all first word vectors and a second vector matrix composed of all second word vectors are obtained respectively.

[0062] Step 62, cosine similarity-based dot product operation is performed on the first vector matrix and the second vector matrix to generate a semantic matching degree data set, wherein the semantic matching degree data set includes document weights and candidate news texts corresponding to each document weight.

[0063] Step 63, based on a set rule, a target document weight is selected from all document weights corresponding to the semantic matching degree data set, and a candidate news text corresponding to the target document weight is taken as a target news text.

[0064] In some optional embodiments, the following steps are adopted: a document weight with the largest weight is selected from all document weights corresponding to the semantic matching degree data set to obtain a target document weight.

[0065] By respectively obtaining the first vector matrix composed of all first word vectors and the second vector matrix composed of all second word vectors in the above steps; performing cosine similarity-based dot product operation on the first vector matrix and the second vector matrix to generate a semantic matching degree data set, wherein the semantic matching degree data set includes document weights and candidate news texts corresponding to each document weight; based on a set rule, a target document weight is selected from all document weights corresponding to the semantic matching degree data set, and a candidate news text corresponding to the target document weight is taken as a target news text, the semantic distance is determined based on the cosine distance, and then the document matching degree is obtained, and the news required by the user to search is determined according to the document matching degree.

[0066] In some embodiments, the hierarchical index data set is constructed, including the following steps:

[0067] Step 71, natural language processing is performed on news text data crawled from a preset data source to obtain corresponding news keywords, wherein the preset data source is determined based on domain name information in a set web domain name library, the natural language processing includes text cleaning, entity recognition and keyword extraction, and the news keywords include entities obtained by entity recognition and keywords obtained by keyword extraction.

[0068] Step 72, based on the news text data and the corresponding news keywords, a news keyword index and a keyword news index are respectively constructed, and a preset vectorization processing mode is used to perform vectorization processing on the news keywords to generate a word vector index.

[0069] Step 73, all word vectors in the word vector index are processed by a preset clustering algorithm to obtain a clustered keyword index.

[0070] In this embodiment, the method of clustering is obtained by a keyword vector clustering algorithm, wherein the clustering algorithm includes but is not limited to a K-means clustering algorithm.

[0071] Step 74, based on the word vector of the news keyword corresponding to each cluster in the cluster keyword index, determine the center point word vector corresponding to each cluster, and take the center point word vector as the word cluster vector of the corresponding cluster, and generate a word cluster vector index with all clusters and the word cluster vector corresponding to each cluster.

[0072] In this embodiment, the generation of the word vector can adopt one of the following ways: using a pre-trained language model, using vector training, calling an open source vector; the word cluster vector corresponding to the cluster is the center point word vector in the word vector of all news keywords corresponding to the cluster.

[0073] Figure 2 is a schematic diagram of a semantic retrieval system according to the preferred embodiment of the present application; Figure 3 is a schematic diagram of the formation of a hierarchical index dataset in the preferred embodiment of the present application; Figure 4 is a schematic diagram of the flow of news retrieval in the preferred embodiment of the present application; Figure 5 is a schematic diagram of the operation of calculating keyword clusters in the embodiment of the present application; Figure 6 is a schematic diagram of the operation of calculating text matching degree in the embodiment of the present application. The news retrieval method of the preferred embodiment of the present application is further described as follows with reference to Figures 2 to 6

[0074] The semantic retrieval system of the embodiment of the present application is divided into an offline part and an online part.

[0075] The main functions of the offline part are crawler, document preprocessing and hierarchical index establishment function, and the main tasks of the offline part include web seed management, crawling news data, news preprocessing and index establishment, wherein,

[0076] Web seed management is mainly responsible for seed library and its management, management of the effectiveness of the website and the effectiveness of the web processing template, and modification or change when seed invalidation or web processing template invalidation is found.

[0077] News data crawling is mainly to crawl the seed website at a fixed time and process the web to obtain news.

[0078] News preprocessing involves NLP processing, including text cleaning, entity recognition, and keyword extraction; text cleaning is to remove interfering content in news text, such as advertisements, publishers, and reporter names in news text, which have little or no meaning for news retrieval; entity recognition extracts names, institutions, and places in the article, and the entity recognition method uses the fine-tuning method of the pre-trained language model in deep learning; keyword extraction extracts keywords in the text according to the algorithm, and the algorithm used is the Textrank algorithm. ​

[0079] The index is the focus of offline work, reference Figure 3 , the index to be established in the embodiment of the application is 5, which are: news keyword index, keyword news index, word vector index, clustering keyword index and word clustering vector index.

[0080] The main function of the online part is to extract user query keywords and semantic retrieval.

[0081] Reference Figure 4 , the online retrieval process includes the following 7 steps, specifically:

[0082] Step S1, extracting keywords.

[0083] Step S2, keyword vectorization.

[0084] Step S3, calculating semantic distance with word clustering vector index and taking the first K clusters.

[0085] Step S4, through the query clustering keyword index and keyword news index, candidate news text is obtained.

[0086] Step S5, query the news keyword index to obtain the second keyword.

[0087] Step S6, query the word vector index to obtain the second word vector.

[0088] Step S7, calculating semantic similarity.

[0089] Among them, the calculation process of step S3 is as shown in Figure 5 , which is obtained by matrix operation and then in a certain dimension, the operation complexity of such operation is very low, and the result can be quickly obtained. Specifically, reference Figure 5 , the clustering order can be quickly obtained by cosine similarity, and U and C in the figure are matrices, the superscripts n and m of the matrix are the dimensions of the corresponding matrix, and d is the dimension of the vectorization. avg is the average function, axis is the dimension of the average, matmul is the matrix dot product operation; in this embodiment, since there are only two word vector matrices, the cosine similarity is used for matrix dot product operation, and the difference between the cosine similarity is that there is no denominator part.

[0090] Among them, the calculation process of step S7 is as shown in Figure 6 , the document matching degree can be quickly obtained by keyword vector dot product budget, Figure 6 , and Figure 5The same part of the symbol corresponds to the same meaning of the symbol, and in the figure, U and D are matrices, the upper indexes n and m of the matrix are the dimensions of the corresponding matrix, d is the dimension of the vectorization, avg is the average function, axis is the dimension of the average, matmul is the matrix dot product operation, sum and sqrt are the sum and average functions, respectively.

[0091] In the embodiment, a news retrieval device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and details have been described above. The terms "module", "unit", "sub-unit" and the like used below can be a combination of software and / or hardware that can implement a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware can also be implemented and conceived.

[0092] Figure 7 is a structural block diagram of a news retrieval device provided by an embodiment of the application, as shown in Figure 7 The news retrieval device provided by the embodiment of the application comprises:

[0093] The receiving module 71 is configured to receive a retrieval request of a user and extract a first keyword from the retrieval request, wherein the first keyword is used to represent a target news text required by the user to retrieve.

[0094] The obtaining module 72 is coupled to the receiving module 71 and is configured to obtain a constructed hierarchical index dataset, wherein the hierarchical index dataset comprises a news keyword index, a keyword news index, a word vector index, a clustered keyword index and a word cluster vector index.

[0095] The preliminary screening module 73 is coupled to the obtaining module 72 and is configured to determine a target keyword cluster based on a first word vector corresponding to the first keyword and the word cluster vector index, and sequentially index the clustered keyword index and the keyword news index by taking the target keyword cluster as an initial index value to obtain a plurality of candidate news texts.

[0096] The processing module 74 is coupled to the preliminary screening module 73 and is configured to index a second keyword corresponding to the candidate news text in the news keyword index, and after indexing out a second word vector corresponding to the second keyword from the word vector index, retrieve a target news text from the plurality of candidate news texts based on a semantic similarity between the first word vector and the second word vector.

[0097] The news retrieval device provided by the embodiment of the present application receives a retrieval request of a user, extracts a first keyword from the retrieval request, wherein the first keyword is used to represent a target news text required by the user; acquires a constructed hierarchical index dataset, wherein the hierarchical index dataset comprises a news keyword index, a keyword news index, a word vector index, a clustered keyword index and a word cluster vector index; determines a target keyword cluster based on a first word vector corresponding to the first keyword and the word cluster vector index, and indexes the clustered keyword index and the keyword news index in turn with the target keyword cluster as an initial index value to obtain a plurality of candidate news texts; indexes a second keyword corresponding to the candidate news text in the news keyword index, indexes out a second word vector corresponding to the second keyword from the word vector index, and retrieves the target news text from the plurality of candidate news texts based on semantic similarity of the first word vector and the second word vector, thereby solving the problem that a semantic retrieval system in the related art cannot realize fast semantic retrieval under the condition of limited computing resources, and achieving the beneficial effects of fast semantic retrieval under low resource conditions, semantic matching of the semantic retrieval system and balance of speed.

[0098] In some embodiments, the preliminary screening module 73 further comprises:

[0099] The first acquisition unit is configured to acquire a word vector matrix composed of all the first word vectors and a word cluster vector matrix composed of all the word cluster vectors in the word cluster vector index, respectively.

[0100] The first calculation unit is coupled to the first acquisition unit and is configured to perform a matrix dot product operation based on a class cosine similarity on the word vector matrix and the word cluster vector matrix to generate a keyword cluster set, wherein the keyword cluster set comprises keyword clusters and cluster weights corresponding to the keyword clusters.

[0101] The first screening unit is configured to select a preset number of keyword clusters from the keyword cluster set based on the cluster weights and a preset rule to obtain the target keyword cluster.

[0102] In some embodiments, the first screening unit is further configured to select a preset number of keyword clusters from the keyword cluster set in descending order of the cluster weights to obtain the target keyword cluster.

[0103] In some embodiments, the preliminary screening module 73 further comprises:

[0104] The second obtaining unit obtains a clustering keyword index and a keyword news index. The clustering keyword index includes a cluster, an index keyword, and relationship information between the cluster and the index keyword. The keyword news index includes the index keyword, a first news text, and relationship information between the index keyword and the first news text.

[0105] The first querying unit is coupled to the second obtaining unit and is configured to query, in the clustering keyword index, an index keyword corresponding to a target keyword cluster, and query, in the keyword news index, a first news text corresponding to the index keyword, to obtain a plurality of candidate news texts.

[0106] In some embodiments, the processing module 74 is further configured to obtain a news keyword index, wherein the news keyword index includes a second news text, a news keyword, and relationship information between the second news text and the news keyword; query, in the news keyword index, a news keyword corresponding to the candidate news text taken as the second news text, and take the news keyword as a second keyword corresponding to the candidate news text.

[0107] In some embodiments, the processing module 74 is further configured to obtain a keyword vector index, wherein the keyword vector index includes a candidate keyword, a keyword vector, and relationship information between the candidate keyword and the keyword vector; query, in the keyword vector index, a keyword vector corresponding to the second keyword taken as the candidate keyword, and take the keyword vector as a second keyword vector corresponding to the second keyword.

[0108] In some embodiments, the processing module 74 is further configured to respectively obtain a first vector matrix composed of all first keyword vectors and a second vector matrix composed of all second keyword vectors; perform a dot product operation based on cosine similarity on the first vector matrix and the second vector matrix to generate a semantic matching degree data set, wherein the semantic matching degree data set includes a document weight and a candidate news text corresponding to each document weight; select a target document weight from all document weights corresponding to the semantic matching degree data set based on a set rule, and take a candidate news text corresponding to the target document weight as a target news text.

[0109] In some embodiments, the processing module 74 is further configured to select a document weight with the largest weight from all document weights corresponding to the semantic matching degree data set to obtain the target document weight.

[0110] In some embodiments, the news retrieval device further includes a construction module, and the construction module includes:

[0111] The first processing unit is configured to perform natural language processing on news text data crawled from a preset data source to obtain corresponding news keywords, wherein the preset data source is determined based on domain name information in a set web domain name library, the natural language processing includes text cleaning, entity recognition and keyword extraction, and the news keywords include entities obtained through entity recognition and keywords obtained through keyword extraction.

[0112] The first construction unit is coupled to the first processing unit and configured to construct the news keyword index and the keyword news index based on the news text data and the corresponding news keywords, respectively, and perform vectorization processing on the news keywords by using a preset vectorization processing mode to generate the keyword vector index.

[0113] The second calculation unit is coupled to the first construction unit and configured to perform clustering processing on all keyword vectors in the keyword vector index by using a preset clustering algorithm to obtain the clustered keyword index.

[0114] The third calculation unit is coupled to the second calculation unit and configured to determine a center point keyword vector corresponding to each cluster based on keyword vectors of the news keywords corresponding to the cluster in the clustered keyword index, take the center point keyword vector as a keyword cluster vector corresponding to the cluster, and generate the keyword cluster vector index based on all clusters and the keyword cluster vector corresponding to each cluster.

[0115] Figure 8 is a structural schematic diagram of an electronic device of an embodiment of the present application, as Figure 8 shown, the present application provides an electronic device, which comprises a processor 81, a communication interface 82, a memory 83 and a communication bus 84, wherein the processor 81, the communication interface 82 and the memory 83 complete communication with each other through the communication bus 84,

[0116] The memory 83 is configured to store a computer program.

[0117] The processor 81 is configured to implement the method steps in the method when executing the program stored in the memory 83. Figure 1

[0118] The processing in the electronic device implements the method steps in the method, and the technical effects brought by the processing are consistent with the technical effects of the method steps in the news retrieval method executed by the above-mentioned embodiment, which will not be described here again. Figure 1 Figure 1

[0119] ​​​The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 8 Only one thick line is used to represent the bus in the above electronic device, but it does not mean that there is only one bus or only one type of bus.

[0120] The communication interface is used for communication between the above electronic device and other devices.

[0121] The memory can include a Random Access Memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the above processor.

[0122] The above processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0123] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the news retrieval method provided by any one of the preceding method embodiments.

[0124] In another embodiment provided by the present application, a computer program product containing instructions is also provided, which, when running on a computer, causes the computer to perform the steps of the news retrieval method described in any one of the above embodiments.

[0125] It has to be noted that, in the present document, the terms "first", "second", etc. merely serve the purpose of differentiation and do not require or imply any kind of ordering or sequence of entities or actions, but are merely used to distinguish a first entity or action from a second entity or action. Furthermore, the terms "comprising", "containing", etc. are meant not to be limiting to the respective processes, methods, articles, or apparatuses, but are meant to be open-ended and do not exclude additional elements or steps. The terms "including", "containing" or any other similar terms are used in the sense of "comprising" and / or "including" and are not used in an excluding or exhaustive sense.

[0126] The foregoing is considered as illustrative only of the principles of the application. Numerous modifications and changes will readily occur to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Accordingly, the scope of the application is indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are intended to be embraced therein.

Claims

1. A news search method characterized by comprising: The method comprises the following steps: receiving a search request of a user, and extracting a first keyword from the search request, wherein the first keyword is used to represent a target news text of the user demand search; obtaining a constructed hierarchical index dataset, wherein the hierarchical index dataset comprises a news keyword index, a keyword news index, a word vector index, a clustered keyword index and a word cluster vector index; determining a target keyword cluster based on a first word vector corresponding to the first keyword and the word cluster vector index, and sequentially indexing the clustered keyword index and the keyword news index with the target keyword cluster as an initial index value to obtain a plurality of candidate news texts; indexing a second keyword corresponding to the candidate news text in the news keyword index, and after indexing a second word vector corresponding to the second keyword from the word vector index, searching the target news text from the plurality of candidate news texts based on semantic similarity of the first word vector and the second word vector; wherein the hierarchical index dataset is constructed by: performing natural language processing on news text data crawled from a preset data source to obtain corresponding news keywords, wherein the preset data source is determined based on domain name information in a set web domain name library, the natural language processing comprises text cleaning, entity recognition and keyword extraction, and the news keywords comprise entities obtained by entity recognition and keywords obtained by keyword extraction; constructing the news keyword index and the keyword news index based on the news text data and the corresponding news keywords, and performing vectorization processing on the news keywords by using a preset vectorization processing mode to generate the word vector index; performing clustering processing on all word vectors in the word vector index by using a preset clustering algorithm to obtain the clustered keyword index; determining a center point word vector corresponding to each cluster based on the word vector of the news keyword corresponding to each cluster in the clustered keyword index, taking the center point word vector as a word cluster vector corresponding to the cluster, and generating the word cluster vector index based on all clusters and the word cluster vector corresponding to each cluster.

2. The method of claim 1, wherein, determining a target keyword cluster based on a first word vector corresponding to the first keyword and the word cluster vector index, comprising: respectively obtaining a word vector matrix composed of all the first word vectors and a word cluster vector matrix composed of all the word cluster vectors in the word cluster vector index; performing matrix point multiplication operation on the word vector matrix and the word cluster vector matrix based on class cosine similarity to generate a keyword cluster set, wherein the keyword cluster set comprises keyword clusters and cluster weights corresponding to the keyword clusters; selecting a preset number of keyword clusters from the keyword cluster set based on the cluster weights and a preset rule to obtain the target keyword cluster.

3. The method of claim 2, wherein, selecting a preset number of keyword clusters from the keyword cluster set based on the cluster weights and a preset rule to obtain the target keyword cluster, comprising: Selecting a preset number of keyword clusters from the keyword cluster set in a descending order of cluster weights to obtain the target keyword cluster.

4. The method of claim 1, wherein, Taking the target keyword cluster as an initial index value, sequentially indexing the cluster keyword index and the keyword news index to obtain a plurality of candidate news texts, including: Obtaining the cluster keyword index and the keyword news index, wherein the cluster keyword index includes clusters, index keywords, and relationship information between the clusters and the index keywords, and the keyword news index includes the index keywords, first news texts, and relationship information between the index keywords and the first news texts; In the cluster keyword index, querying the index keywords corresponding to the target keyword cluster, and in the keyword news index, querying the first news texts corresponding to the index keywords to obtain a plurality of candidate news texts.

5. The method of claim 1, wherein, In the news keyword index, indexing the second keywords corresponding to the candidate news texts, including: Obtaining the news keyword index, wherein the news keyword index includes second news texts, news keywords, and relationship information between the second news texts and the news keywords; Taking the candidate news text as the second news text, querying the corresponding news keyword in the news keyword index, and taking the news keyword as the second keyword corresponding to the candidate news text.

6. The method of claim 5, wherein, Indexing the second keyword vector corresponding to the second keyword from the keyword vector index, including: Obtaining the keyword vector index, wherein the keyword vector index includes alternative keywords, keyword vectors, and relationship information between the alternative keywords and the keyword vectors; Taking the second keyword as the alternative keyword, querying the corresponding keyword vector in the keyword vector index, and taking the keyword vector as the second keyword vector corresponding to the second keyword.

7. The method of claim 1, wherein, Retrieving the target news text from a plurality of candidate news texts based on semantic similarity of the first keyword vector and the second keyword vector, including: Respectively obtaining a first vector matrix composed of all the first keyword vectors and a second vector matrix composed of all the second keyword vectors; Performing a dot product operation based on cosine similarity on the first vector matrix and the second vector matrix to generate a semantic matching degree dataset, wherein the semantic matching degree dataset includes document weights and the candidate news texts corresponding to each document weight; Based on a set rule, selecting a target document weight from all the document weights corresponding to the semantic matching degree dataset, and taking the candidate news text corresponding to the target document weight as the target news text.

8. The method of claim 7, wherein, Based on a set rule, selecting a target document weight from all the document weights corresponding to the semantic matching degree dataset, including: selecting the document weight with the largest weight from all the document weights corresponding to the semantic matching degree dataset to obtain the target document weight.

9. A news search apparatus characterized by comprising: Including: The receiving module is configured to receive a search request of a user and extract a first keyword from the search request, wherein the first keyword is used to represent a target news text of the user demand search; The obtaining module is configured to obtain a constructed hierarchical index dataset, wherein the hierarchical index dataset comprises a news keyword index, a keyword news index, a word vector index, a clustered keyword index, and a word cluster vector index; The preliminary screening module is configured to determine a target keyword cluster based on a first word vector corresponding to the first keyword and the word cluster vector index, and sequentially perform indexing in the clustered keyword index and the keyword news index with the target keyword cluster as an initial index value to obtain a plurality of candidate news texts; The processing module is configured to index a second keyword corresponding to the candidate news text in the news keyword index, and after indexing a second word vector corresponding to the second keyword from the word vector index, search the target news text from the plurality of candidate news texts based on semantic similarity of the first word vector and the second word vector. The hierarchical index dataset is constructed by: performing natural language processing on news text data crawled from a preset data source to obtain corresponding news keywords, wherein the preset data source is determined based on domain name information in a set web domain name library, the natural language processing comprises text cleaning, entity recognition, and keyword extraction, and the news keywords comprise entities obtained through entity recognition and keywords obtained through keyword extraction; constructing the news keyword index and the keyword news index based on the news text data and the corresponding news keywords, and performing vectorization processing on the news keywords using a preset vectorization processing mode to generate the word vector index; performing clustering processing on all word vectors in the word vector index using a preset clustering algorithm to obtain the clustered keyword index; determining a center point word vector corresponding to each cluster based on word vectors of the news keywords corresponding to the cluster, taking the center point word vector as a word cluster vector corresponding to the cluster, and generating the word cluster vector index based on all clusters and the word cluster vector corresponding to each cluster.

10. An electronic device, comprising: The computer program is executed by the processor to implement the steps of the news search method according to any one of claims 1-8. The computer program is executed by the processor to implement the steps of the news search method according to any one of claims 1-8. ​ 11. A computer readable storage medium having stored thereon a computer program, characterized in that, ​

Images