Article retrieval method, program product, equipment and storage medium

Abstract

The embodiment of the invention provides an article retrieval method, a program product, equipment and a storage medium, and the method comprises the steps: obtaining a first target word frequency vector of each article from an article library in response to input retrieval content; the first target word frequency vector is obtained after dimension reduction of the first initial word frequency vector; the first initial word frequency vector of each article is determined based on the occurrence frequency of each segmented word in the article; determining a second initial word frequency vector of the retrieval content based on the occurrence frequency of each segmented word in the retrieval content, and performing dimensionality reduction on the second initial word frequency vector to obtain a second target word frequency vector; the dimension of the second target word frequency vector is consistent with that of the first target word frequency vector; and based on the similarity between each first target word frequency vector and the second target word frequency vector, determining a target article matched with the retrieval content from an article library. The first target word frequency vector greatly reduces the occupation amount of the memory and carries word segmentation word frequency information, so that retrieval and sorting are carried out at the same time.

Description

Technical Field

[0001] The present application relates to the field of retrieval technology, and in particular to an article retrieval method, program product, device and storage medium. Background Art

[0002] In the related art, the traditional full-text search technology uses the reverse index method. Although the reverse index method is simple in principle, when the article involves a large number of words, there will be a huge amount of indexes, and the memory usage is very high. At the same time, the reverse index can only determine that the retrieved article includes the segmented words in the retrieval content, but such retrieval results cannot meet other user needs such as sorting based on word frequency. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an article retrieval method, program product, device and storage medium to achieve the technical effect of reducing memory usage and improving user retrieval experience.

[0004] The first aspect of the present application embodiment provides an article retrieval method, the method comprising: In response to the input search content, a first target word frequency vector corresponding to each article is obtained from a preset article library; wherein the first target word frequency vector is obtained by reducing the dimension of the first initial word frequency vector; the first initial word frequency vector corresponding to each article is determined based on the number of occurrences of each word in the preset word segmentation dictionary in the article; Based on the number of occurrences of each word segment in the word segmentation dictionary in the search content, determine a second initial word frequency vector of the search content, and perform dimension reduction on the second initial word frequency vector to obtain a second target word frequency vector; the dimension of the second target word frequency vector is consistent with that of the first target word frequency vector; Based on the similarity between each of the first target word frequency vectors and the second target word frequency vectors, a target article matching the search content is determined from the article library.

[0005] In the above implementation process, the first target word frequency vector carries the word segmentation information in the article and the number of times the word segmentation appears in the article. At the same time, the memory usage of the first target word frequency vector after dimensionality reduction is greatly reduced. During the retrieval process, the matching target article is found by calculating the similarity between the first target word frequency vector and the second target word frequency vector, thereby converting part of the storage pressure into computing pressure. Moreover, there is no need to create an additional retrieval library to implement full-text retrieval. It is only necessary to perform word frequency statistics on the original articles in the article library, and the adaptation requirements for the database are relatively low. At the same time, since the first target word frequency vector can represent the number of times a word segmentation appears in the article, retrieval and sorting can be carried out simultaneously without the need to introduce other sorting algorithms, thereby improving the user's retrieval experience.

[0006] Furthermore, the word segmentation dictionary includes multiple word segmentation sets; the position of each word segmentation in the word segmentation set in the word segmentation dictionary is related to the semantics of the word segmentation, and the position distance of multiple word segments in the word segmentation set in the word segmentation dictionary is negatively correlated with the semantic similarity between the multiple word segments.

[0007] In the above implementation process, by dividing the word segmentation set, multiple word segmentations in the same field or multiple word segmentations with similar semantics are divided into the same word segmentation set, and the positions of multiple word segmentations in each word segmentation set are determined based on semantic relevance, so that the semantically similar word segmentations in the word segmentation set are closer, which is conducive to the retention of text features in subsequent dimensionality reduction operations.

[0008] Furthermore, the word segmentation dictionary is constructed by the following steps: Obtaining the segmented words included in all articles in the article library, and clustering the segmented words to obtain multiple cluster centers and a segmented word set corresponding to each cluster center; Determine the interval position of each word segmentation set in the word segmentation dictionary; For each word segmentation set, evaluating the semantic similarity between multiple word segments in the word segmentation set; The position of each word in the word set in the interval position is determined based on the semantic similarity to obtain a word segmentation dictionary.

[0009] In the above implementation process, by semantically clustering the segmented words, the segmented words involved in the article are divided into multiple segmented word sets, and then the semantically similar segmented words in the segmented word sets are set in similar positions, which is conducive to retaining text feature information in subsequent dimensionality reduction operations.

[0010] Furthermore, the elements of each dimension in the first target word frequency vector are obtained by the following steps: Obtain a first initial word frequency vector; the first initial word frequency vector includes a plurality of first sub-vectors, the first sub-vectors corresponding to the word segmentation set; the first sub-vector includes a plurality of elements, the number of elements in the first sub-vector is consistent with the number of word segmentations included in the word segmentation set; For each target dimension in each second sub-vector, obtain the weight of each word segmentation under the target dimension, and perform weighted summation on the corresponding elements in the first sub-vector based on the weight corresponding to each word segmentation to obtain the elements of the target dimension; wherein the second sub-vector is obtained after dimensionality reduction of the first sub-vector; and the weight of each word segmentation under the first target dimension is positively correlated with the importance of the word segmentation in the word segmentation set.

[0011] In the above implementation process, by distributing the weights of the word segmentation in each dimension of the second sub-vector, and at the same time distributing the weights of the word segmentation that is important to the word segmentation set mainly to the first target dimension, the elements under the first target dimension can retain more semantic information of the word segmentation with high importance, thus avoiding the loss of semantic information of important word segmentation during dimensionality reduction. In addition, by arranging the word segmentation with similar semantics in similar positions in the word segmentation dictionary, the first target word frequency vector after dimensionality reduction can retain the semantic information of the word segmentation to a greater extent, thus reducing the loss of semantic information caused by dimensionality reduction.

[0012] Furthermore, the first initial word frequency vector is normalized.

[0013] In the above implementation process, normalizing the first initial word frequency vector can improve the distribution characteristics of the first initial word frequency vector and enhance the dimensionality reduction effect.

[0014] Furthermore, the determining, from the article library, a target article matching the search content based on the similarity between the first target word frequency vector and the second target word frequency vector includes: Determine a target subvector corresponding to the target field to which the search content belongs in each of the first target word frequency vectors; For each first target word frequency vector, increase the target sub-vector according to a preset domain weight factor to obtain a weighted first target word frequency vector; Based on the similarity between the weighted first target word frequency vector and the second target word frequency vector, a target article matching the search content is determined from the article library.

[0015] In the above implementation process, since the target sub-vector is enlarged in the weighted first target word frequency vector, the role of word segmentation in the target field in the similarity calculation is strengthened, and the role of word segmentation in other fields in the similarity calculation is weakened. The similarity result obtained will be more affected by the target field, effectively reducing the interference of high-frequency word segmentation in other fields, thereby improving the retrieval accuracy.

[0016] Furthermore, the determining, from the article library, a target article matching the search content based on the similarity between the first target word frequency vector and the second target word frequency vector includes: Merge each of the first target word frequency vectors into a word frequency matrix; Parallel calculation of the dot product of the first target word frequency matrix and the second target word frequency vector in each row of the word frequency matrix, and determination of the calculation result as the similarity; A target article matching the search content is determined from the article library based on the similarity.

[0017] In the above implementation process, the first target word frequency vector of each article is merged into a word frequency matrix. Matrix operations do not require serial operations, and parallel computing resources can be used, such as the parallel logic of the GPU to implement matrix operations, thereby comprehensively improving the similarity calculation speed, improving retrieval efficiency, and making full use of computing resources.

[0018] A second aspect of an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, any method described in the first aspect is implemented.

[0019] According to a third aspect of the present application, an electronic device is provided, the electronic device comprising: processor; a memory for storing processor-executable instructions; Wherein, when the processor calls the executable instruction, it implements the operation of any method described in the first aspect.

[0020] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A schematic diagram of a flow chart of an article retrieval method provided in an embodiment of the present application; Figure 2-Figure 3 A schematic diagram of another article retrieval method provided in an embodiment of the present application; Figure 4 A schematic diagram of dimensionality reduction processing of the first initial word frequency vector provided in an embodiment of the present application; Figure 5-Figure 6 A schematic diagram of another article retrieval method provided in an embodiment of the present application; Figure 7 A hardware structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0024] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0025] The reverse indexing method refers to the process of indexing articles with each word in the article as an index to point to the article after the article is segmented, so as to establish an index library. For example, if article A contains segmentation 1 and segmentation 3, and article B contains segmentation 2 and segmentation 3, then segmentation 1 is indexed to article A, segmentation 2 is indexed to article B, and segmentation 3 is indexed to article A and article B. When performing a full-text search later, the input search content is segmented, and the full-text search function can be realized by finding the articles involved in each segmentation through the segmentation index of the search content. Although the reverse indexing method is simple in principle, when the article vocabulary is large, the search method with segmentation as the index will have the problem of huge index volume, which takes up a lot of memory. At the same time, the reverse index can only determine that the retrieved article includes the segmentation in the search content, and cannot count the number of times the segmentation appears in the article. If the user needs to sort the search results by word frequency, it is necessary to re-count the word frequency of the articles in the search results for sorting, or introduce other sorting algorithms. Therefore, the reverse indexing method cannot meet other user needs such as sorting based on word frequency.

[0026] In order to solve at least one of the above technical problems, the present application provides an article retrieval method for performing full-text retrieval on all articles in a preset article library. Specifically, it includes: Figure 1 Steps 110 to 130 are shown.

[0027] Step 110: In response to the input search content, a first target word frequency vector corresponding to each article is obtained from a preset article library.

[0028] Among them, the first target word frequency vector is obtained by reducing the dimension of the first initial word frequency vector; the first initial word frequency vector corresponding to each of the articles is determined based on the number of occurrences of each word in the preset word segmentation dictionary in the article.

[0029] Exemplarily, the article library includes multiple articles and a first target word frequency vector corresponding to each article. There is a one-to-one correspondence between the articles and the first target word frequency vector. The first target word frequency vector is obtained by reducing the dimension of the first initial word frequency vector. There is also a one-to-one correspondence between the first initial word frequency vector and the articles.

[0030] In addition, a word segmentation dictionary is pre-established, which includes the word segmentations involved in all articles in the article library. The first initial word frequency vector corresponding to each article is determined based on the number of occurrences of each word segmentation in the word segmentation dictionary in the article. The dimension of the first initial word frequency vector is consistent with the number of word segmentations included in the word segmentation dictionary. It can be seen that the first word frequency vector corresponding to an article can not only represent the word segmentations included in the article, but also represent the number of occurrences of the word segmentation in the article.

[0031] For example, if the word segmentation dictionary includes m word segmentations, namely a, b, ...m, and article 1 includes word segmentations a, b and m, and word segmentation a appears once, word segmentation b appears 5 times, and word segmentation m appears 3 times, then the first initial word frequency vector is an m-dimensional vector and can be expressed as (1, 5, 0, 0, ...3).

[0032] It can be seen that before executing the retrieval method, the first initial word frequency vector corresponding to each article is generated in advance, and then the first initial word frequency vector corresponding to each article is reduced in dimension to obtain the first target word frequency vector. The dimension reduction process of the first target word frequency vector will be unfolded below. The first target word frequency vector is stored in the article library and an association relationship is established with the corresponding article. In subsequent use, it is possible to periodically monitor whether new articles are added to the article library, and automatically update the word segmentation dictionary and the newly added corresponding first target word frequency vector for the newly added articles to maintain the timeliness and accuracy of the retrieval algorithm.

[0033] Thus, when executing step 110, when the input search content is received, such as the search content input by the user, the first target word frequency vector corresponding to each article is obtained from the article library.

[0034] Step 120: Based on the number of occurrences of each word in the word segmentation dictionary in the search content, determine a second initial word frequency vector of the search content, and reduce the dimension of the second initial word frequency vector to obtain a second target word frequency vector; the dimension of the second target word frequency vector is consistent with that of the first target word frequency vector.

[0035] Exemplarily, the input search content can be first segmented to determine the segmented words included in the search content. The segmented words used may include but are not limited to the JIEBA algorithm, the THULAC algorithm, and SnowNLP, etc. Then, based on the number of occurrences of each segmented word in the search content in the segmented word dictionary, the second initial word frequency vector of the search content is determined. Then, the second initial word frequency vector is reduced in dimension to obtain the second target word frequency vector of the search content, and the second target word frequency vector has the same dimension as the first target word frequency vector, for example, the first target word frequency vector and the second target word frequency vector are both n-dimensional vectors. The dimensionality reduction process of the second target word frequency vector will be described below.

[0036] Step 130: Based on the similarity between each of the first target word frequency vectors and the second target word frequency vectors, determine a target article matching the search content from the article library.

[0037] Exemplarily, the similarity between the first target word frequency vector and the second target word frequency vector corresponding to each article can be calculated respectively. For example, the similarity between the first target word frequency vector and the second target word frequency vector can be characterized by, but not limited to, cosine similarity, Euclidean distance, Manhattan distance, etc. Then, based on the similarity, a target article matching the search content is determined from the article library. For example, the article corresponding to the first target word frequency vector with the highest similarity can be determined as the target article. For another example, the article corresponding to the first target word frequency vector with a similarity higher than a preset similarity threshold can be determined as the target article.

[0038] It is understandable that the traditional reverse index requires the establishment of an index relationship between the number of segmentations and the article corresponding to the number of segmentations involved in the article, and it is impossible to reduce the dimension, resulting in a large amount of memory usage. However, the present application determines the first initial word frequency vector of the article based on the number of occurrences of each segmentation in the article in the segmentation dictionary, and then reduces the dimension of the first initial word frequency vector to obtain the first target word frequency vector, so that the first target word frequency vector carries both the segmentation information in the article and the number of occurrences of the segmentation in the article. The elements of each dimension in the first target word frequency vector can establish a connection with the segmentation, and at the same time, the memory usage of the first target word frequency vector after dimensionality reduction is greatly reduced. During the retrieval process, the matching target article is found by calculating the similarity between the first target word frequency vector and the second target word frequency vector, thereby converting part of the storage pressure into computing pressure. Moreover, there is no need to create an additional retrieval library to achieve full-text retrieval. It is only necessary to perform word frequency statistics on the original articles in the article library, and the adaptation requirements for the database are relatively low. At the same time, since the first target word frequency vector can represent the number of times a word appears in an article, retrieval and sorting can be performed simultaneously without the need to introduce other sorting algorithms, thereby improving the user's retrieval experience.

[0039] The following is a detailed introduction to step 110 to step 130.

[0040] Regarding the word segmentation dictionary, in some embodiments, the word segmentation dictionary includes multiple word segmentation sets. Each word segmentation set corresponds to a different interval position in the word segmentation dictionary. Among them, different word segmentation sets can correspond to vocabulary in different fields. The fields include but are not limited to the medical field, the catering field, the construction field, etc.

[0041] The position of each word in the word segmentation set in the word segmentation dictionary, or the specific position in the interval position, is related to the semantics of the word segmentation. Specifically, for each word segmentation set, the position distance of multiple word segments in the word segmentation set in the word segmentation dictionary, or the position distance in the interval position, is negatively correlated with the semantic relevance between the multiple word segments. That is, if the semantic relevance of two word segments is higher, the position distance of the two word segments in the word segmentation dictionary is smaller, and the two word segments are closer.

[0042] Optionally, the segmentation set may include a sensitive segmentation set, and the sensitive segmentation set may include sensitive segmentations. Placing the sensitive segmentation set at a specific position in the segmentation dictionary facilitates shielding the search for sensitive segmentations when needed.

[0043] It can be seen that this embodiment divides the word segmentation set so that multiple word segmentations in the same field or multiple word segmentations with similar semantics are divided into the same word segmentation set, and the positions of multiple word segmentations in each word segmentation set are determined based on semantic relevance, so that the semantically similar word segmentations in the word segmentation set are closer, which is beneficial to the retention of text features in subsequent dimensionality reduction operations.

[0044] According to some embodiments of the present application, the word segmentation dictionary can be Figure 2 Steps 210 - 240 are shown constructed.

[0045] Step 210: Obtain the segmented words included in all articles in the article library, and cluster the segmented words to obtain multiple cluster centers and a segmented word set corresponding to each cluster center.

[0046] For example, each article in the article library can be firstly segmented to obtain all the segmented words included in all articles. Then all the segmented words are clustered. The clustering of the segmented words can be based on semantics. The specific clustering process can be referred to the relevant technology, which will not be expanded in this embodiment. After the clustering is completed, multiple clustering centers and the segmented word sets corresponding to each clustering center can be obtained. For example, one or more segmented word sets correspond to the vocabulary of a field.

[0047] Step 220: Determine the interval position of each word segmentation set in the word segmentation dictionary.

[0048] Exemplarily, after obtaining a plurality of word segmentation sets, the interval position corresponding to each word segmentation set in the word segmentation dictionary may be determined, so that each word segmentation set in the word segmentation dictionary is independent of each other.

[0049] Step 230: For each word segmentation set, evaluate the semantic similarity between multiple word segments in the word segmentation set.

[0050] For example, for each word segmentation set, the semantic similarity between every two word segmentations in the word segmentation set can be evaluated separately. The specific evaluation method of the semantic similarity can be referred to in the related art, which will not be elaborated here.

[0051] Step 240: Determine the position of each word in the word set in the interval position based on the semantic similarity to obtain a word segmentation dictionary.

[0052] Finally, the position of each segmentation in the interval is determined based on the semantic similarity between multiple segmentations in the segmentation set, and the segmentation dictionary is obtained by arranging them.

[0053] It can be seen that this embodiment divides the segmentations involved in the article into multiple segmentation sets by semantically clustering the segmentations, and then sets the semantically similar segmentations in the segmentation sets at similar positions, which is conducive to retaining text feature information in subsequent dimensionality reduction operations.

[0054] Based on any of the above embodiments, the elements of each dimension in the first target word frequency vector are expressed as follows: Figure 3 The steps 310-320 shown are obtained.

[0055] Step 310: Obtain a first initial word frequency vector, wherein the first initial word frequency vector includes a plurality of first sub-vectors, the first sub-vectors corresponding to the word segmentation set; the first sub-vectors include a plurality of elements, and the number of elements in the first sub-vectors is consistent with the number of word segmentations included in the word segmentation set.

[0056] As described above, since the word segmentation dictionary includes multiple word segmentation sets, the first initial word frequency vector can also be divided into multiple first sub-vectors. The first sub-vectors and the word segmentation sets are in a one-to-one correspondence. Each first sub-vector includes one or more elements, so the number of elements in each first sub-vector is consistent with the number of word segmentations included in the word segmentation set corresponding to the first sub-vector.

[0057] For example, Figure 4 As shown, if the word segmentation dictionary includes word segmentation set A and word segmentation set B, where word segmentation set A includes word segmentation a1 to word segmentation a i There are i word segments in total, and the word segmentation set B includes word segments b1 to b j There are j word segments in total, so the first initial word frequency vector of each article is an (i+j)-dimensional vector, such as Figure 4 As shown. And the first initial word frequency vector includes the first sub-vector With the first subvector The first subvector Corresponding to the word segmentation set A, it includes i elements, the first sub-vector Corresponding to the word segmentation set B, it includes j elements.

[0058] Step 320: For each target dimension in each second sub-vector, obtain the weight of each word segment in the target dimension, and perform weighted summation on the corresponding elements in the first sub-vector based on the weight corresponding to each word segment to obtain the elements of the target dimension. The second sub-vector is obtained by reducing the dimension of the first sub-vector; the weight of each word segment in the first dimension is positively correlated with the importance of the word segment in the word segment set.

[0059] It can be understood that each first sub-vector in the first initial word frequency vector is reduced in dimension separately without interfering with each other. In this way, each first sub-vector is reduced in dimension to obtain a second sub-vector. The first target word frequency vector includes multiple second sub-vectors. Then, when calculating the elements of each target dimension in each second sub-vector, first determine the word segmentation set corresponding to the corresponding first sub-vector, and then obtain the weight of each word segmentation in the word segmentation set under the target dimension. Then, based on the weight corresponding to each word segmentation, the corresponding elements in the first sub-vector are weighted summed to obtain the elements of the target dimension.

[0060] Continue to see Figure 4, the first target word frequency vector includes multiple second sub-vectors, and each second sub-vector is obtained by reducing the dimension of the corresponding first sub-vector in the first initial word frequency vector. is the first subvector Dimensionality reduction is obtained, where p<i; the second subvector is the first subvector Dimensionality reduction yields , where q<j.

[0061] With the second subvector For example, the second subvector is a p-dimensional vector with a total of p elements. For the second subvector The first target dimension in T 11 The dimension in which it is located, first determine the first subvector The corresponding word set A, and then obtain the weight of each word in the word set A under the first target dimension, that is, the weight w of word a1 under the first target dimension 11 、……participle a i The weight w in the first target dimension 1i . Then the first sub-vector is weighted The corresponding elements in are weighted summed to obtain the elements of the first target dimension, that is, Similarly, the element of the pth target dimension , and so on.

[0062] It can be seen that if the rth first subvector in the first initial word frequency vector includes v elements, then the element T of the sth target dimension in the rth second subvector in the first target word frequency vector is rs The calculation formula is: .

[0063] In addition, the weight of each segmentation under the first target dimension is positively correlated with the importance of the segmentation in the segmentation set. The importance refers to the degree of correlation between the segmentation and the field indicated by the segmentation set. For example, if a segmentation set indicates the medical field, the segmentations "vaccine" and "pneumonia" have a higher importance in the segmentation set, and the corresponding weight under the first target dimension is also higher. In addition, the sum of the weights of the segmentations under all target dimensions is 1. It can be seen that by distributing the weights of the segmentations in each dimension of the second sub-vector, and at the same time distributing the weights of the segmentations that are important to the segmentation set mainly to the first target dimension, the elements under the first target dimension can retain more semantic information of the segmentations with high importance, avoiding the loss of semantic information of important segmentations during dimensionality reduction. In addition, by arranging semantically similar segmentations in similar positions in the segmentation dictionary, the first target word frequency vector after dimensionality reduction can retain the semantic information of the segmentations to a greater extent, reducing the loss of semantic information caused by dimensionality reduction.

[0064] In addition, according to some embodiments of the present application, before the first initial word frequency vector is subjected to dimensionality reduction, the first initial word frequency vector may be normalized to improve the distribution characteristics of the first initial word frequency vector and enhance the dimensionality reduction effect.

[0065] In addition, the generation process of the second initial word frequency vector and the dimensionality reduction process of the second initial word frequency vector can refer to the dimensionality reduction process of the first initial word frequency vector, and this application will not be repeated here.

[0066] Based on any of the above embodiments, step 130 may specifically include: Figure 5 Steps 510 - 530 are shown.

[0067] Step 510: Determine the target sub-vector corresponding to the target field to which the search content belongs in each of the first target word frequency vectors.

[0068] Exemplarily, after obtaining the input search content, the field to which the search content belongs can be determined. The method of determining the field to which the search content belongs can be referred to the relevant technology, and this embodiment will not be expanded here. As described above, one or more word segmentation sets in the word segmentation dictionary can refer to a field, so after determining the target field to which the search content belongs, the target sub-vector corresponding to the target field can be determined in each first target word frequency vector. In addition, since the first target word frequency vector includes multiple second sub-vectors, the target sub-vector is actually one or more of the second sub-vectors.

[0069] Step 520: For each first target word frequency vector, increase the target sub-vector according to a preset domain weight factor to obtain a weighted first target word frequency vector.

[0070] Exemplarily, after determining the target sub-vector, each element in the target sub-vector may be dot-multiplied by a preset domain weight factor, thereby increasing the target sub-vector to obtain a weighted first target word frequency vector. The domain weight factor is greater than 1.

[0071] Step 530: Based on the similarity between the weighted first target word frequency vector and the second target word frequency vector, determine a target article matching the search content from the article library.

[0072] It can be seen that since the target sub-vector is enlarged in the weighted first target word frequency vector, the role of word segmentation in the target field in the similarity calculation is strengthened, and the role of word segmentation in other fields in the similarity calculation is weakened. The similarity result obtained will be more affected by the target field, effectively reducing the interference of high-frequency word segmentation in other fields, thereby improving the retrieval accuracy.

[0073] In addition, based on any of the above embodiments, step 130 may specifically include: Figure 6 Steps 610 - 630 are shown.

[0074] Step 610: Merge each of the first target word frequency vectors into a word frequency matrix.

[0075] Exemplarily, each first target word frequency vector can be used as a row vector in a word frequency matrix, thereby obtaining a word frequency matrix composed of multiple first target word frequency vectors.

[0076] Step 620: Calculate in parallel the dot product of the first target word frequency matrix and the second target word frequency vector in each row of the word frequency matrix, and determine the calculation result as the similarity.

[0077] Exemplarily, the second target word frequency vector can be a column vector, and the multiplication of the word frequency matrix and the second target word frequency vector is actually a dot product operation of the first target word frequency vector and the second target word frequency vector in each row of the word frequency matrix. In this way, the dot product of the first target word frequency matrix and the second target word frequency vector in each row of the word frequency matrix can be calculated in parallel to obtain a similarity vector. The similarity vector is a column vector, and the elements it includes are similarities.

[0078] Optionally, the parallel logic of a GPU (Graphics Processing Unit) may be used to execute step 620 , thereby fully utilizing computing resources.

[0079] Step 630: Determine a target article matching the search content from the article library based on the similarity.

[0080] Finally, the article corresponding to the first target word frequency vector with the highest similarity can be determined as the target article. For another example, the article corresponding to the first target word frequency vector with a similarity higher than a preset similarity threshold can be determined as the target article.

[0081] It can be seen that this embodiment merges the first target word frequency vector of each article into a word frequency matrix. Matrix operations do not require serial operations, and parallel computing resources can be used, such as the parallel logic of the GPU to implement matrix operations, thereby comprehensively improving the similarity calculation speed, improving retrieval efficiency, and making full use of computing resources.

[0082] In order to better understand this solution, an example is given below to illustrate it.

[0083] Before performing a search, pre-build an article library and a word segmentation dictionary. Specifically, you can count all the word segments involved in all articles in the article library and build a word segmentation dictionary. For example, the article library includes three articles, the content of article A is "test case", the content of article B is "this article is a test case", and the content of article C is "article article". By segmenting the three articles, you can build a word segmentation dictionary as (1. test 2. case 3. this 4. article 5. for).

[0084] Then, based on the number of occurrences of each word in the word segmentation dictionary in the article, the first initial word frequency vector of each article is determined. For example, the first initial word frequency vector of article A is (1, 1, 0, 0, 0), the first initial word frequency vector of article B is (1, 1, 1, 1, 1), and the first initial word frequency vector of article C is (0, 0, 0, 2, 0).

[0085] Then the first initial word frequency vector of each article is normalized so that the modulus of all first initial word frequency vectors is 1. The normalized first initial word frequency vector of article A is (0.707, 0.707, 0, 0, 0), the normalized first initial word frequency vector of article B is (0.447, 0.447, 0.447, 0.447, 0.447), and the normalized first initial word frequency vector of article C is (0, 0, 0, 1, 0).

[0086] In this example, since the word segmentation dictionary has only 5 words, the storage and computing pressure is not large, so dimensionality reduction is not required. However, in actual use, the word segmentation dictionary usually has hundreds of thousands of words, and the first initial word frequency vector has hundreds of thousands of dimensions, so a dimensionality reduction algorithm is needed to reduce the dimension of the first initial word frequency vector to reduce storage and computing pressure.

[0087] When starting a search, all the word frequency vectors in the article library (in this example, the first initial word frequency vector, if dimensionality reduction is performed, the first target word frequency vector) can be merged into a word frequency matrix, which is shown below:

[0088] Get the input search content "article", and determine the second initial word frequency vector of the search content as (0, 0, 0, 1, 0) based on the number of occurrences of each word in the word segmentation dictionary in the search content. Similarly, in actual use, the second initial word frequency vector needs to be normalized and reduced in dimension. However, in this example, since the dimensionality reduction process is omitted, the word frequency matrix and the second initial word frequency vector are subsequently multiplied by a dot multiplication operation. The operation process is as follows:

[0089] This gives us the similarity vector (0, 0.447, 1) T , each element in the similarity vector is the similarity between the corresponding article and the search content. That is, the similarity between the search content "article" and article A is 0, the similarity with article B is 0.447, and the similarity with article C is 1. Finally, according to the similarity results, the target articles matching the search content are obtained, and multiple target articles are arranged in descending order of similarity, so as to realize the full-text search function of retrieval and sorting.

[0090] Based on the article retrieval method described in any of the above embodiments, the present application also provides a computer program product, which includes one or more computer programs or instructions. The computer program or instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. When the computer program is executed by a processor, an article retrieval method described in any of the above embodiments is implemented.

[0091] Based on an article retrieval method described in any of the above embodiments, the present application also provides Figure 7 A schematic diagram of the structure of an electronic device is shown in FIG. Figure 7 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement an article retrieval method described in any of the above embodiments.

[0092] The present application also provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, it can be used to execute an article retrieval method described in any of the above embodiments.

[0093] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0094] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0095] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0096] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0097] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0098] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. An article retrieval method, characterized in that: The method comprises: In response to the input search content, a first target word frequency vector corresponding to each article is obtained from a preset article library; wherein the first target word frequency vector is obtained by reducing the dimension of the first initial word frequency vector; the first initial word frequency vector corresponding to each article is determined based on the number of occurrences of each word in the preset word segmentation dictionary in the article; Based on the number of occurrences of each word segment in the word segmentation dictionary in the search content, determine a second initial word frequency vector of the search content, and perform dimension reduction on the second initial word frequency vector to obtain a second target word frequency vector; the dimension of the second target word frequency vector is consistent with that of the first target word frequency vector; Based on the similarity between each of the first target word frequency vectors and the second target word frequency vectors, a target article matching the search content is determined from the article library.

2. The method according to claim 1, characterized in that The word segmentation dictionary includes multiple word segmentation sets; the position of each word segmentation in the word segmentation dictionary is related to the semantics of the word segmentation, and the position distance of multiple word segments in the word segmentation set in the word segmentation dictionary is negatively correlated with the semantic similarity between the multiple word segments.

3. The method according to claim 2, characterized in that The word segmentation dictionary is constructed by the following steps: Obtaining the segmented words included in all articles in the article library, and clustering the segmented words to obtain multiple cluster centers and a segmented word set corresponding to each cluster center; Determine the interval position of each word segmentation set in the word segmentation dictionary; For each word segmentation set, evaluating the semantic similarity between multiple word segments in the word segmentation set; The position of each word in the word set in the interval position is determined based on the semantic similarity to obtain a word segmentation dictionary.

4. The method according to claim 2 or 3, characterized in that: The elements of each dimension in the first target word frequency vector are obtained by the following steps: Obtain a first initial word frequency vector; the first initial word frequency vector includes a plurality of first sub-vectors, the first sub-vectors corresponding to the word segmentation set; the first sub-vector includes a plurality of elements, the number of elements in the first sub-vector is consistent with the number of word segmentations included in the word segmentation set; For each target dimension in each second sub-vector, obtain the weight of each word segmentation under the target dimension, and perform weighted summation on the corresponding elements in the first sub-vector based on the weight corresponding to each word segmentation to obtain the elements of the target dimension; wherein the second sub-vector is obtained after dimensionality reduction of the first sub-vector; and the weight of each word segmentation under the first target dimension is positively correlated with the importance of the word segmentation in the word segmentation set.

5. The method according to claim 4, characterized in that The first initial word frequency vector is normalized.

6. The method according to claim 2, characterized in that The determining, from the article library, a target article matching the search content based on the similarity between the first target word frequency vector and the second target word frequency vector comprises: Determine a target subvector corresponding to the target field to which the search content belongs in each of the first target word frequency vectors; For each first target word frequency vector, increase the target sub-vector according to a preset domain weight factor to obtain a weighted first target word frequency vector; Based on the similarity between the weighted first target word frequency vector and the second target word frequency vector, a target article matching the search content is determined from the article library.

7. The method according to claim 1 or 6, characterized in that: The determining, from the article library, a target article matching the search content based on the similarity between the first target word frequency vector and the second target word frequency vector comprises: Merge each of the first target word frequency vectors into a word frequency matrix; Parallel calculation of the dot product of the first target word frequency matrix and the second target word frequency vector in each row of the word frequency matrix, and determination of the calculation result as the similarity; A target article matching the search content is determined from the article library based on the similarity.

8. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, when the processor calls the executable instructions, the operation of any method described in claims 1-7 is implemented.

10. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed by a processor, the steps of any method described in claims 1-7 are implemented.

Images