Method, device, equipment and medium for calculating the correlation between search text and library files
By considering the importance of word segmentation and the domain importance of the library file in the correlation calculation of search text and library text, and establishing a reverse set in advance, the problems of inaccurate and low stability of correlation calculation in the prior art are solved, and the problem of inaccurate correlation calculation and low stability are realized. More efficient and accurate correlation calculations.
Patent Information
- Application Number
- CN201910604516.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-07-05
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-08-02
AI Technical Summary
When calculating the correlation between search text and library text, the prior art does not fully consider the importance of word segmentation in search text, and is greatly affected by the length of library text, resulting in inaccurate correlation calculation results and low stability.
By obtaining search text, perform word segmentation operations, calculate the word weight of each word segmentation, and calculate the correlation degree based on the importance parameters in the library file, reduce the impact of library text length on the results, and establish a reversed set in advance to reduce the irrelevant library file calculate.
It significantly improves the accuracy and speed of correlation calculations and improves the stability of calculation results.
Smart Images

Figure CN110347795B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of correlation calculation, and in particular to a method, device, equipment and medium for calculating the correlation between a search text and a library file. Background Art
[0002] In the prior art, the relevance of a search text to each library text in a library text set is usually calculated based on a probabilistic retrieval model. Specifically, the relevance of a search text to the library text can be calculated based on the relevance of each word segment of the search text to the library text and the general importance of each word segment, wherein the relevance of the word segment to the library text is easily affected by the length of each library text in the library text set.
[0003] In the prior art, the importance of word segmentation in the search text is not fully considered when calculating the relevance, and it is greatly affected by the length of each library text in the library text set. If the length of each library text in the library text set varies greatly, the stability of the relevance result between the search text and the library text obtained based on the prior art is also low, resulting in inaccurate relevance calculation results. Summary of the invention
[0004] In order to solve the technical problem that the calculation accuracy of the relevance between a search text and a library file is not high in the prior art, an embodiment of the present invention provides a method, device, equipment and medium for calculating the relevance between a search text and a library file.
[0005] In one aspect, the present invention provides a method for calculating the relevance between a search text and a library file, the method comprising:
[0006] Acquire a search text, and perform a word segmentation operation on the search text to obtain a target word segmentation set, wherein each target word segmentation in the target word segmentation set points to a word weight parameter corresponding to the target word segmentation;
[0007] Calculate the importance parameter of each word in the target word set according to the library file set where the library file is located;
[0008] Obtain a target library text collection for relevance calculation;
[0009] The relevance between the search text and each target library text in the target library text set is calculated according to the target word segmentation set and the importance parameter of each word in the target word segmentation set.
[0010] On the other hand, the present invention provides a device for calculating the relevance between a search text and a library file, the device comprising:
[0011] A search text acquisition module is used to acquire a search text and perform a word segmentation operation on the search text to obtain a target word segmentation set, wherein each target word segmentation in the target word segmentation set points to a word weight parameter corresponding to the target word segmentation;
[0012] An importance parameter acquisition module, used for calculating the importance parameter of each word in the target word set according to the library file set where the library file is located;
[0013] A target library text set acquisition module is used to acquire a target library text set for performing relevance calculation;
[0014] The relevance calculation module is used to calculate the relevance between the search text and each target library text in the target library text set according to the target word set and the importance parameter of each word in the target word set.
[0015] On the other hand, the present invention provides a device, characterized in that the device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement a method for calculating the relevance between a search text and a library file.
[0016] On the other hand, the present invention provides a computer storage medium, characterized in that the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by a processor and executes a method for calculating the correlation between a search text and a library file.
[0017] The present invention provides a method, device, equipment and medium for calculating the relevance between a search text and a library file. The present invention comprehensively considers the importance of each word segment in the search text and the importance of different fields in the library file when calculating the relevance, and reduces the influence of the length of the library text on the relevance calculation, thereby significantly improving the accuracy of the relevance calculation. Furthermore, the embodiment of the present invention avoids calculating the relevance of library files that are completely unrelated to the search text by pre-establishing an inverted set, thereby reducing the amount of relevance calculation and improving the relevance calculation speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 It is a flow chart of a method for calculating the relevance between a search text and a library file provided by the present invention;
[0020] Figure 2 It is a schematic diagram of an implementation environment of a method for calculating the relevance between a search text and a library file provided by the present invention;
[0021] Figure 3 It is a flow chart of a method for calculating the relevance between a search text and a library file provided by the present invention;
[0022] Figure 4 The present invention provides a flow chart of performing a word segmentation operation on the text to obtain a target word segmentation set, wherein each target word segmentation in the target word segmentation set points to a word weight parameter corresponding to the target word segmentation;
[0023] Figure 5 The present invention provides a flowchart for obtaining a target library text set for performing relevance calculation;
[0024] Figure 6 It is a flow chart of the inverted set of the calculation library text set for each template word segmentation provided by the present invention;
[0025] Figure 7 The present invention provides a flowchart for calculating the relevance between the search text and each target library text in the target library text set according to the target word set and the importance parameter of each word in the target word set;
[0026] Figure 8 is a flowchart of related data processing after the relevance is obtained provided by the present invention;
[0027] Fig. 9 It is a schematic diagram showing click-through rate performance in search products of the embodiment of the present invention and the prior art provided by the present invention;
[0028] Fig.10 The present invention provides a block diagram of a device for calculating the relevance between a search text and a library file;
[0029] Fig.11 It is a block diagram of a target library text set acquisition module provided by the present invention;
[0030] Fig.12 It is a schematic diagram of the operating environment of a device for calculating the relevance between a search text and a library file provided by the present invention;
[0031] Fig.13 The present invention provides a schematic diagram of the hardware structure of a device for implementing the method provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.
[0034] In order to make the purpose, technical solution and advantages disclosed in the embodiments of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present invention and are not used to limit the embodiments of the present invention. In order to explain the embodiments of the present invention in detail, in the embodiments of the present invention, the related concepts involved are first explained:
[0035] IDF: Inverse Dcument Frequency, inverse text frequency. Used to measure the general importance of a word.
[0036] TF: Term Frequency. It is used to represent the frequency of a word appearing in a certain file.
[0037] TW: Term Weight, used to represent the importance of a word in a text.
[0038] In order to elaborate on the embodiment of the present invention, the embodiment of the present invention first describes the prior art solution for calculating the relevance between the search text and the library file. Figure 1 As shown, the calculation scheme includes:
[0039] S1. Perform a word segmentation operation on the search text to obtain at least one word segmentation.
[0040] S3. Obtain the relevance score between each word segment and the library file.
[0041] Specifically, you can refer to the formula Calculate the correlation between the word segmentation and the library file, where q i ,d,R(q i ,d) respectively represent the score of word segmentation, library file and the correlation between word segmentation and library file, f i represents the frequency of the word segmentation in the library file, dl, avgdl represent the length of the library file and the average length of all library files in the library file set where the library file is located respectively. k1, b are adjustment factors, generally k1∈[1,3], b∈[0.5,1]. According to the formula, the higher the frequency of the word segmentation in the library file, the higher the relevance score between the word segmentation and the library file, the higher the ratio of the length of the library file to the average length of all library files in the library file set, and the lower the relevance score between the word segmentation and the library file.
[0042] S5. Obtain the inverse text frequency of each word segment.
[0043] Specifically, you can refer to the formula Calculate the inverse text frequency of the word segmentation, where IDF(q i ),n(q i ), N respectively represent the inverse text frequency of the word segmentation, the number of library files in the library file set where the library file is located that hit the word segmentation, and the total number of library files in the library file set where the library file is located. According to the formula, the more common the word, the lower its corresponding inverse text frequency, and the rarer the word, the higher its corresponding inverse text frequency.
[0044] S7. Taking the inverse text frequency corresponding to the word segmentation as the weight and the correlation score between the word segmentation and the library file as the addend, the correlation between the search text and the library file is calculated based on the weighted average method.
[0045] According to the description of the above-mentioned prior art, the calculation of relevance in the prior art does not take into account the hit position of the word in the library file, the importance of the word in the search text, and is greatly affected by the length of each library text in the library text set, thereby affecting the accuracy of the relevance calculation.
[0046] In order to obtain a more accurate relevance calculation result, an embodiment of the present invention provides a method for calculating the relevance between a search text and a library file. First, an embodiment of the present invention discloses an implementation scenario of the method for calculating the relevance between a search text and a library file.
[0047] See also Figure 2The implementation environment includes: a client 01 and a server 03, wherein the client 01 and the server 03 are in communication connection, the client 01 obtains the search text and transmits the search text to the server 03, the server 03 queries the relevance of the library file in the library stored therein for calculating the relevance to the search text and the search text, and interacts with the client 01 according to the calculation result of the relevance.
[0048] The client 01 can communicate with the server 03 based on the browser / server mode (B / S) or the client / server mode (C / S). The client 01 can include physical devices such as smart phones, tablet computers, laptops, digital assistants, smart wearable devices, and vehicle terminals, and can also include software running in physical devices, such as applications. For example, the client 01 can run search software.
[0049] The server 03 may include an independently operated server, or a distributed server, or a server cluster consisting of multiple servers.
[0050] The embodiment of the present invention provides a method for calculating the relevance between a search text and a library file. Figure 3 As shown, the method includes:
[0051] S101. Acquire a search text, and perform a word segmentation operation on the search text to obtain a target word segmentation set, wherein each target word segmentation in the target word segmentation set points to a word weight parameter corresponding to the target word segmentation.
[0052] Specifically, the word segmentation operation can adopt a Chinese character string matching word segmentation method, a word meaning word segmentation method or a statistical word segmentation method. The Chinese character string matching word segmentation method can be a forward maximum matching method, a reverse maximum matching method, a bidirectional maximum matching method or a shortest path word segmentation method, etc. The embodiment of the present invention does not limit the specific word segmentation operation method.
[0053] The word weight parameter is used to characterize the weight of the target word in the search text. The word weight parameter may be related to factors such as the number of characters in the target word and the search popularity of the target word.
[0054] Specifically, in a feasible implementation, the text is segmented to obtain a target segmentation set, and each target segmentation in the target segmentation set points to a word weight parameter corresponding to the target segmentation, such as Figure 4 As shown, including:
[0055] S1011. Perform word segmentation processing on the search text to obtain at least one word segmentation.
[0056] S1013. Obtain the word weight correlation corresponding to each word segment based on the preset corpus.
[0057] The word weight related vector may include one or more combinations of the length of the longest corpus segment that matches the word segment with the corpus, the number of times the word segment is hit by the corpus in the corpus, the average length of the corpus that hits the word segment, and the length of the shortest corpus that hits the word segment in the corpus.
[0058] S1015. Determine the word weight parameters corresponding to each word segment by using the word weight correlation and the pre-constructed word weight parameter calculation model.
[0059] Specifically, the word weight parameter calculation model can be obtained by training based on an existing neural network model.
[0060] S1017. Construct a target word segmentation set based on each word segment and its corresponding word weight parameter.
[0061] S103. Calculate the importance parameter of each word in the target word set according to the library file set in which the library file is located.
[0062] Specifically, the importance parameter can be represented by the inverse text frequency. The importance parameter can refer to the formula Where IDF(q i ),n(q i ), N respectively represent the inverse text frequency of the word segmentation, the number of library files in the library file set where the library file is located that hit the word segmentation, and the total number of library files in the library file set where the library file is located.
[0063] S105. Obtain a target library text set for relevance calculation.
[0064] In the prior art, the target library text set is the complete set of the library text set where the library text is located, that is, in the prior art, the relevance between each library text in the library text set and the search text is calculated. The embodiment of the present invention believes that not every library text will hit at least one word in the search text, so the library files in the library text set that do not hit the search text do not need to participate in the relevance calculation. Therefore, the embodiment of the present invention adds a step of filtering the library files in the library text set. The target library text set obtained in this embodiment can be a subset or the complete set of the library text set.
[0065] Specifically, the target library text set for relevance calculation is obtained, such as Figure 5 As shown, including:
[0066] S1051. Calculate the inverted set of the library text set for each template word segment.
[0067] In the prior art, the process of searching for records based on non-primary attributes is called inverted indexing. In the embodiment of the present invention, the significance of obtaining an inverted index set is that the relevant library texts that hit the segmentation can be queried based on the segmentation.
[0068] In order to facilitate the subsequent online use of inverted sets, in the embodiment of the present invention, inverted sets are constructed for as many template word segments as possible, so that they cover the word segments in the search text as much as possible.
[0069] Specifically, the inverted set of the library text set for each template word segment is calculated, such as Figure 6 Shown include:
[0070] S10511. Obtain the domain of each library text in the library text collection.
[0071] Specifically, the library text may include multiple fields, such as title, introduction, body, etc.
[0072] S10513. Obtain the word frequency of each of the template word segments relative to the library text, wherein the word frequency of the template word segment relative to the library text is the maximum value of the word frequency of the template word segment relative to the text content in each domain of the library text.
[0073] Specifically, the word frequency of the text content in each domain of the library text can be calculated by referring to the formula (ab*field_length)*field_weight, where a and b are adjustment factors, field_length and field_weight are the number of times the template segmentation word appears in the text content in the domain, and the importance of the domain.
[0074] S10515. Construct an inverted set for each of the template word segments, wherein the word frequency of the template word segment relative to any library text in the inverted set is greater than a preset threshold.
[0075] S1053. Record the mapping relationship between each template word segment and its corresponding inverted set to obtain an inverted dictionary.
[0076] In a feasible embodiment, the steps of constructing the inverted set corresponding to each word segment and recording the mapping relationship between each word segment and its corresponding inverted set can be completed in the data preprocessing stage or offline, and the mapping relationship is shown in Table 1. Taking the online search scenario as an example, in the offline state, the background server can construct an inverted set for all recognizable word segments, store the inverted set, and record the mapping relationship between each word segment and its corresponding inverted set. When the user enters the search text online, the library text related to the search text can be obtained according to the inverted set constructed offline, and only the correlation between the library text related to the search text and the search text is calculated, thereby improving the calculation efficiency.
[0077] Table 1
[0078]
[0079] S1055. Obtain a target inverted set corresponding to each target word in the target word set according to the inverted dictionary.
[0080] S1057. Take the union of the target inverted sets corresponding to each target word segment as the target library text set.
[0081] Specifically, if the search text is an Eastern European patent, the target word segmentation set includes two word segmentations, "Eastern Europe" and "patent". As shown in Table 1, the inverted set corresponding to "Eastern Europe" includes four library texts, the inverted set corresponding to "patent" includes two library texts, and the target library text set includes six library texts.
[0082] Compared with the prior art in which relevance calculation is performed on each library text in a library text set, the technical solution described in the embodiment of the present invention can only perform relevance calculation on six library texts in the target library text set. Obviously, the relevance calculation speed is greatly improved and the amount of relevance calculation is significantly reduced.
[0083] S107. Calculate the relevance between the search text and each target library text in the target library text set according to the target word segmentation set and the importance parameter of each word in the target word segmentation set.
[0084] Specifically, the relevance between the search text and each target library text in the target library text set is calculated based on the target word set and the importance parameter of each word in the target word set, such as Figure 7 As shown, including:
[0085] S1071. Calculate the importance of the search text relative to each target library text in the target library text set.
[0086] The importance of the search text relative to a target library text can be calculated according to the formula To calculate, where TW(q i ),IDF(q i ) represents the word weight parameter and importance parameter of the target word segmentation. If i∈(q∩d), then q i Indicates the word segmentation that is hit by both the searched text and the target library text. If i∈q, then q i Indicates the target word hit by the search text.
[0087] S1073. Calculate the distance between the search text and each target library text in the target library text set.
[0088] The distance between the search text and a target library text can be calculated according to the formula To calculate, where tf(q i ) represents the frequency of the target word segment relative to the target library text. It can be calculated in step 1051 and stored in the inverted dictionary constructed in step 1053.
[0089] S1075. Calculate the relevance of the search text to each target library text in the target library text set according to the importance of the search text relative to each target library text in the target library text set and the distance between the search text and each target library text in the target library text set.
[0090] Specifically, in the embodiment of the present invention, the product of the importance of the search text and a certain target library text and the distance between the search text and the target library text can be used as the relevance between the search text and the target library text.
[0091] In other feasible embodiments, after obtaining the result of the correlation, such as Figure 8 As shown, it may also include:
[0092] S201. Arrange the target library texts in descending order of relevance to obtain a target library text sorting table.
[0093] S203: Select the first N target library texts in the target library text sorting table as output texts.
[0094] Specifically, the value of N can be set according to actual needs. The output text can be used as a rendering basis or a recall basis.
[0095] The method for calculating the relevance between a search text and a library file disclosed in an embodiment of the present invention comprehensively considers the importance of each word segment in the search text and the importance of different fields in the library file when calculating the relevance, and reduces the influence of the length of the library text on the relevance calculation, thereby significantly improving the accuracy of the relevance calculation. Furthermore, the embodiment of the present invention avoids calculating the relevance of library files that are completely unrelated to the search text by pre-establishing an inverted set, thereby reducing the amount of relevance calculation and improving the relevance calculation speed.
[0096] The embodiments of the present invention can be widely used in scenarios where relevance calculation is required. For example, in a search product, the method provided in the embodiments of the present invention can be used to directly calculate the relevance between the search text and the library file after obtaining the search text, and the search results can be displayed based on the relevance calculation results. If the search product supports voice input, the user's input voice can also be obtained and converted into search text, and then the relevance calculation method disclosed in the embodiments of the present invention can be used to obtain the final search results. Please refer to Fig. 9 , which shows a schematic diagram of the click-through rate performance of the embodiment of the present invention and the prior art in the search product. The schematic diagram uses the horizontal axis to represent time and the vertical axis to represent the click-through rate, and each broken line corresponds to 10% of the traffic. In the schematic diagram, at the time node represented by the vertical black line, the multiple broken lines with higher vertical coordinates all use the relevance calculation method described in this application, and the multiple broken lines with lower vertical coordinates use the prior art. Obviously, the click-through rate of the embodiment of the present invention is significantly improved compared with the prior art, which is due to the improved accuracy of the relevance calculation of the embodiment of the present invention.
[0097] The embodiment of the present invention also includes a device for calculating the relevance between a search text and a library file, such as Fig.10 As shown, the device comprises:
[0098] A search text acquisition module 301 is used to acquire a search text and perform a segmentation operation on the search text to obtain a target segmentation set, wherein each target segmentation in the target segmentation set points to a word weight parameter corresponding to the target segmentation;
[0099] The importance parameter acquisition module 303 is used to calculate the importance parameter of each word in the target word set according to the library file set where the library file is located;
[0100] A target library text set acquisition module 305 is used to acquire a target library text set for performing relevance calculation;
[0101] The relevance calculation module 307 is used to calculate the relevance between the search text and each target library text in the target library text set according to the target word set and the importance parameter of each word in the target word set.
[0102] Specifically, Fig.11 As shown, the target library text set acquisition module 305 includes:
[0103] An inverted set calculation unit 3051 is used to calculate the inverted set of the library text set for each template word segment;
[0104] An inverted dictionary acquisition module 3053 is used to record the mapping relationship between each template word segment and its corresponding inverted set to obtain an inverted dictionary;
[0105] A target inverted set acquisition unit 3055 is used to acquire a target inverted set corresponding to each target word in the target word set according to the inverted dictionary;
[0106] The target library text set acquisition unit 3057 is used to take the union of the target inverted sets corresponding to each target word segment as the target library text set.
[0107] In the embodiment of the present invention, Fig.12 As shown, the inverted set calculation unit 3051 and the inverted dictionary acquisition module 3053 can run in an offline environment, while the search text acquisition module 301, the importance parameter acquisition module 303, the target inverted set acquisition unit 3055 and the target library text set acquisition unit 3057 all run in an online environment.
[0108] Specifically, the device and method for calculating the relevance between a search text and a library file described in the embodiments of the present invention are based on the same inventive concept.
[0109] An embodiment of the present invention further provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor and executing a method for calculating the relevance between a search text and a library file described in an embodiment of the present invention.
[0110] Specifically, the instructions include:
[0111] Acquire a search text, and perform a word segmentation operation on the search text to obtain a target word segmentation set, wherein each target word segmentation in the target word segmentation set points to a word weight parameter corresponding to the target word segmentation;
[0112] Calculate the importance parameter of each word in the target word set according to the library file set where the library file is located;
[0113] Obtain a target library text collection for relevance calculation;
[0114] The relevance between the search text and each target library text in the target library text set is calculated according to the target word segmentation set and the importance parameter of each word in the target word segmentation set.
[0115] Furthermore, the word segmentation operation is performed on the text to obtain a target word segmentation set, wherein each target word segmentation in the target word segmentation set points to a word weight parameter corresponding to the target word segmentation, including:
[0116] Performing word segmentation processing on the search text to obtain at least one word segmentation;
[0117] Based on the preset corpus, the word weight correlation corresponding to each word segmentation is obtained;
[0118] Determine the word weight parameters corresponding to each word segment by using the word weight correlation and the pre-constructed word weight parameter calculation model;
[0119] Construct the target word segmentation set according to each word segmentation and its corresponding word weight parameter.
[0120] Furthermore, the step of obtaining a target library text set for performing relevance calculation includes:
[0121] Calculate the inverted set of the library text set for each template word segmentation;
[0122] Record the mapping relationship between each template word segment and its corresponding inverted set to obtain an inverted dictionary;
[0123] Acquire a target inverted set corresponding to each target word in the target word set according to the inverted dictionary;
[0124] The union of the target inverted sets corresponding to each target word segmentation is taken as the target library text set.
[0125] Furthermore, the calculation library text set for each template word segmentation inverted set includes:
[0126] Get the domain of each library text in the library text collection;
[0127] Obtaining the word frequency of each of the template segmentations relative to the library text, wherein the word frequency of the template segmentation relative to the library text is the maximum value of the word frequency of the template segmentation relative to the text content in each domain of the library text;
[0128] An inverted set of each template word segment is constructed, wherein the word frequency of the template word segment relative to any library text in the inverted set is greater than a preset threshold.
[0129] Further, the calculating the relevance between the search text and each target library text in the target library text set according to the target word set and the importance parameter of each word in the target word set includes:
[0130] Calculating the importance of the search text relative to each target library text in the target library text set;
[0131] Calculating the distance between the search text and each target library text in the target library text set;
[0132] The relevance of the search text to each target library text in the target library text set is calculated according to the importance of the search text relative to each target library text in the target library text set and the distance between the search text and each target library text in the target library text set.
[0133] Further, the relevance between the search text and each target library text in the target library text set is calculated according to the target word set and the importance parameter of each word in the target word set, and then it also includes:
[0134] Arrange the target library texts in descending order of relevance to obtain a target library text ranking table;
[0135] The first N target library texts in the target library text sorting table are selected as output texts.
[0136] Further, Fig.13 A schematic diagram of the hardware structure of a device for implementing the method provided by an embodiment of the present invention is shown, and the device may participate in constituting or include the apparatus or system provided by an embodiment of the present invention. Fig.13 As shown, the device 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Fig.13 The structure shown is for illustration only and does not limit the structure of the electronic device. Fig.13 More or fewer components as shown, or with Fig.13 Different configurations are shown.
[0137] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the device 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0138] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, to implement the above-mentioned method for searching the correlation between text and library files. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0139] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the device 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.
[0140] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of device 10 (or mobile device).
[0141] It should be noted that the sequence of the embodiments of the present invention described above is for description only and does not represent the advantages and disadvantages of the embodiments. The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0142] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0143] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for calculating the relevance between a search text and a library file, characterized in that: The method comprises: Acquire a search text, perform a word segmentation operation on the search text to obtain a target word segmentation set, wherein each target word segmentation in the target word segmentation set points to a word weight parameter corresponding to the target word segmentation, and the word weight parameter represents the weight of the target word segmentation in the search text; Calculate the importance parameter of each target word segment according to the library file set where the library file is located, wherein the importance parameter is determined based on the number of library files in the library file set that hit the target word segment and the total number of library files in the library file set; Obtain the domain of each library text, the domain includes title, introduction, and text; take the maximum value of (ab*field_length)*field_weight as the word frequency of the corresponding template segmentation relative to the library text, a and b are adjustment factors, field_length and field_weight are the number of times the template segmentation appears in the corresponding domain and the importance of the domain respectively; construct an inverted set of each template segmentation, and the word frequency of the template segmentation relative to any library text in the inverted set is greater than a preset threshold; record the mapping relationship between each template segmentation and its corresponding inverted set to obtain an inverted dictionary; obtain the target inverted set corresponding to each target segmentation in the target segmentation set according to the inverted dictionary; take the union of the target inverted sets corresponding to each target segmentation as the target library text set; According to the formula Calculate the importance of the search text relative to the target library text, TW(q i ),IDF(q i ) represents the word weight parameter and importance parameter of the target word segmentation; according to the formula Calculate the distance between the search text and the target library text, tf(q i ) represents the frequency of the target word relative to the target library text; if i∈(q∩d), q i Indicates the word segmentation that is hit by both the searched text and the target library text. If i∈q, q i Indicates the target word hit by the search text; The relevance between the search text and the target library text is calculated according to the importance and the distance.
2. The method according to claim 1, characterized in that: The word segmentation operation is performed on the search text to obtain a target word segmentation set, wherein each target word segmentation in the target word segmentation set points to a word weight parameter corresponding to the target word segmentation, including: Performing word segmentation processing on the search text to obtain at least one word segmentation; Based on the preset corpus, the word weight correlation corresponding to each word segmentation is obtained; Determine the word weight parameters corresponding to each word segment by using the word weight correlation and the pre-constructed word weight parameter calculation model; Construct the target word segmentation set according to each word segmentation and its corresponding word weight parameter.
3. The method according to claim 1, characterized in that After calculating the relevance between the search text and the target library text according to the importance and the distance, the method further comprises: Arrange the target library texts in descending order of relevance to obtain a target library text ranking table; The first N target library texts in the target library text sorting table are selected as output texts.
4. A device for calculating the relevance between a search text and a library file, characterized in that: The device comprises: A search text acquisition module is used to acquire a search text, perform a word segmentation operation on the search text to obtain a target word segmentation set, each target word segmentation in the target word segmentation set points to a word weight parameter corresponding to the target word segmentation, and the word weight parameter represents the weight of the target word segmentation in the search text; An importance parameter acquisition module, used for calculating the importance parameter of each target word segmentation according to the library file set in which the library file is located, wherein the importance parameter is determined based on the number of library files in the library file set that hit the target word segmentation and the total number of library files in the library file set; The target library text set acquisition module is used to acquire the domain of each library text, the domain includes title, introduction, and text; the maximum value of (ab*field_length)*field_weight is used as the word frequency of the corresponding template segmentation relative to the library text, a and b are adjustment factors, field_length and field_weight are the number of times the template segmentation appears in the corresponding domain and the importance of the domain respectively; an inverted set of each template segmentation is constructed, and the word frequency of the template segmentation relative to any library text in the inverted set is greater than a preset threshold; the mapping relationship between each template segmentation and its corresponding inverted set is recorded to obtain an inverted dictionary; the target inverted set corresponding to each target segmentation in the target segmentation set is obtained according to the inverted dictionary; the union of the target inverted sets corresponding to each target segmentation is used as the target library text set; Correlation calculation module, used to calculate the Calculate the importance of the search text relative to the target library text, TW(q i ),IDF(q i ) represents the word weight parameter and importance parameter of the target word segmentation; according to the formula Calculate the distance between the search text and the target library text, tf(q i ) represents the frequency of the target word relative to the target library text; if i∈(q∩d), q i Indicates the word segmentation that is hit by both the searched text and the target library text. If i∈q, q i Indicates the target word hit by the search text; and calculates the relevance between the search text and the target library text according to the importance and the distance.
5. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a method for calculating the relevance between a search text and a library file as described in any one of claims 1-3.
6. A computer storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor according to a method for calculating the correlation between a search text and a library file as described in any one of claims 1-3.
