Text matching method and device based on probability distribution, equipment and storage medium
By obtaining semantic feature distributions from a professional knowledge base, calculating the probability distribution of user input text, and calculating similarity distances, the problem of text matching accuracy under the influence of noise in large language models is solved, and stable semantic retrieval in high-noise environments is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUBEI TAIYUE SATELLITE TECH DEV CO LTD
- Filing Date
- 2025-08-07
- Publication Date
- 2026-05-12
AI Technical Summary
In large language models, when user input contains noise (such as spelling errors or grammatical confusion), the accuracy of existing text matching methods decreases and their fault tolerance is low.
By obtaining semantic feature distributions from a professional knowledge base, calculating the probability distribution of user input text, and using formulas such as Bach distance to calculate similarity distance, the text matching results are determined.
It significantly improves the fault tolerance and accuracy of text matching, and can stably retrieve semantically relevant professional knowledge in high-noise environments.
Smart Images
Figure CN121029964B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of natural language processing technology, and in particular to text matching methods, apparatus, devices and storage media based on probability distribution. Background Technology
[0002] In the application of large language models based on the RAG (Retrieval-augmented Generation) method, in order to improve the generation effect of the model in specific domains or professional tasks, the most relevant content is usually retrieved from the professional knowledge base after the user inputs the text, and then submitted to the large language model for processing along with the user input text.
[0003] However, the matching accuracy of such methods often drops significantly when there are errors (such as spelling errors, key term errors, grammatical confusion) or noise (such as irrelevant information, vague expressions) in the user input.
[0004] Therefore, how to improve the fault tolerance of text matching when user input information is noisy is a problem that urgently needs to be solved. Summary of the Invention
[0005] The main objective of this application is to provide a text matching method, apparatus, device, and storage medium based on probability distribution, aiming to solve the technical problem of low fault tolerance in text matching when user input information contains noise.
[0006] To achieve the above objectives, this application proposes a text matching method based on probability distribution, the method comprising:
[0007] Semantic feature distributions of various professional knowledge texts are obtained from the professional knowledge base to obtain a set of knowledge probability distributions;
[0008] Obtain the user input text and calculate the corresponding user text probability distribution;
[0009] Calculate the similarity distance between the user text probability distribution and each distribution in the knowledge probability distribution set;
[0010] The text matching result is determined based on the minimum similarity distance.
[0011] In one embodiment, the step of obtaining the semantic feature distribution of each professional knowledge text from the professional knowledge base to obtain a set of knowledge probability distributions includes:
[0012] Obtain various professional knowledge texts from the professional knowledge base, and calculate the word frequency or word vector of the professional knowledge texts;
[0013] The probability distribution of knowledge text is calculated based on word frequency or word vectors to obtain a set of knowledge probability distributions.
[0014] In one embodiment, the step of calculating the knowledge text probability distribution based on word frequency or word vectors to obtain a knowledge probability distribution set includes:
[0015] If the probability distribution of knowledge texts is calculated based on word vectors, and each knowledge text in the professional knowledge base is numbered sequentially, a set of knowledge numbers is obtained. The set of knowledge numbers includes the mapping relationship between each knowledge text and each knowledge number.
[0016] Select the target knowledge text and vectorize it to obtain word vectors;
[0017] Based on the probability distribution formula, the probability distribution of the target knowledge text is calculated for each dimension of the word vector.
[0018] Establish a mapping relationship between each knowledge number and the probability distribution of the target knowledge text to obtain a set of knowledge probability distributions.
[0019] In one embodiment, the step of selecting target knowledge text includes:
[0020] If this is the first time selecting, the knowledge text corresponding to the starting number in the knowledge number set will be selected as the target knowledge text.
[0021] If this is not the first selection, the current number will be incremented sequentially to become the target number, and the knowledge text corresponding to the target number will be selected as the target knowledge text.
[0022] In one embodiment, the step of establishing the mapping relationship between each knowledge number and the probability distribution of the target knowledge text to obtain a set of knowledge probability distributions includes:
[0023] Traverse the knowledge ID set; if there is unprocessed knowledge text, define the unprocessed knowledge text as the target knowledge text.
[0024] Return to the step of vectorizing the target knowledge text to obtain word vectors.
[0025] In one embodiment, the step of calculating the knowledge text probability distribution based on word frequency or word vectors to obtain a knowledge probability distribution set includes:
[0026] If the probability distribution of knowledge texts is calculated based on word frequency, each knowledge text is obtained from the knowledge base, and the professional vocabulary of each knowledge text is extracted to obtain the word frequency of the professional vocabulary;
[0027] Based on the word frequency of the aforementioned professional terms, inverse document frequency is introduced to calculate the weight of the professional terms, and a weighted calculation result is obtained;
[0028] The weighted calculation results are normalized to obtain a set of knowledge probability distributions.
[0029] In one embodiment, the step of calculating the similarity distance between the user text probability distribution and each distribution in the knowledge probability distribution set includes:
[0030] The similarity distance between the user text probability distribution and each distribution in the knowledge probability distribution set is calculated using the Bach distance formula.
[0031] The formula for the Bach distance is expressed as follows:
[0032]
[0033] in, Indicates the first The probability distribution of a knowledge text. This represents the probability distribution of the user's input text. Indicates the first The similarity distance between the probability distribution of a knowledge text and the probability distribution of the user input text.
[0034] Furthermore, to achieve the above objectives, this application also proposes a text matching device based on probability distribution, the text matching device based on probability distribution comprising:
[0035] The first probability calculation module is used to obtain the semantic feature distribution of each professional knowledge text from the professional knowledge base and obtain a set of knowledge probability distributions.
[0036] The second probability calculation module is used to obtain user input text and calculate the corresponding user text probability distribution.
[0037] The similarity distance calculation module is used to calculate the similarity distance between the user text probability distribution and each distribution in the knowledge probability distribution set;
[0038] The text matching determination module is used to determine the text matching result based on the minimum similarity distance.
[0039] Furthermore, to achieve the above objectives, this application also proposes a probability distribution-based text matching device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the probability distribution-based text matching method described above.
[0040] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the probability distribution-based text matching method described above.
[0041] One or more technical solutions proposed in this application have at least the following technical effects:
[0042] By extracting the semantic feature distributions of various professional knowledge texts from the professional knowledge base, a set of knowledge probability distributions is obtained, solving the problem of failing to capture semantic uncertainty due to the use of fixed word vectors. By abstracting the inherent features of the text through probability distributions, a standardized representation of the knowledge base is achieved.
[0043] The system acquires user input text and calculates the corresponding user text probability distribution. By employing the same method for calculating the user text probability distribution, it overcomes the limitation of word vectors being easily distorted when user input is incorrect. Through a probability distribution model, it captures the core query intent rather than surface-level words, significantly improving the error tolerance of text matching.
[0044] By calculating the similarity distance between the probability distribution of user text and each distribution in the knowledge probability distribution set, the problem of vector similarity calculation being sensitive to small changes is solved. By measuring the overall semantic similarity through distribution distance rather than local matching, the comparison process can tolerate distribution shifts.
[0045] Finally, the text matching result is determined based on the minimum similarity distance, which solves the problem of low matching accuracy of traditional technology, achieves stable retrieval of semantically relevant professional knowledge under high noise, and significantly improves fault tolerance and matching accuracy. Attached Figure Description
[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0048] Figure 1 This is a flowchart illustrating the first embodiment of the text matching method based on probability distribution of this application;
[0049] Figure 2 This is a schematic diagram of a sub-process for calculating the probability distribution of user text in an embodiment of this application;
[0050] Figure 3 This is a flowchart illustrating the second embodiment of the text matching method based on probability distribution of this application;
[0051] Figure 4 This is a schematic diagram of the process for obtaining the knowledge probability distribution set based on the BERT model in the embodiments of this application;
[0052] Figure 5 This is a schematic diagram of a sub-process of step S40 in an embodiment of this application;
[0053] Figure 6 This is a schematic diagram of the module structure of a text matching device based on probability distribution according to an embodiment of this application;
[0054] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the text matching method based on probability distribution in the embodiments of this application.
[0055] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0056] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0057] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0058] In RAG-based applications of large language models, a key strategy to improve the model's generation performance in specific domains or professional tasks is to first retrieve the most semantically relevant content fragments (knowledge entries) from a professional knowledge base (such as documents, knowledge graphs, etc.) when processing user input text. Then, the retrieved professional content is concatenated (or combined in other ways) with the original user input text and submitted to the large language model for processing, thereby leveraging professional knowledge to guide and constrain the generation process.
[0059] The core of current mainstream retrieval methods lies in calculating the similarity (such as cosine similarity) between the user's query vector and all knowledge item vectors, and selecting one or more items with the highest similarity as the result. However, this method has a significant limitation: its matching accuracy is highly dependent on the accuracy of the user's input text. When there are errors in the user's input (such as spelling errors, key term errors, grammatical confusion) or noise (such as irrelevant information, vague expressions), the generated query vector often deviates from its true semantic intent. Because vector similarity calculation is extremely sensitive to this semantic shift, the retrieved "most relevant" professional knowledge items may deviate significantly from, or even be completely irrelevant to, the content actually needed by the user. This phenomenon is particularly common in large-scale application scenarios because the quality of user input varies greatly, making it difficult to guarantee accuracy.
[0060] Therefore, how to improve the fault tolerance of text matching when user input information is noisy is a problem that urgently needs to be solved.
[0061] Based on this, embodiments of this application provide a text matching method based on probability distribution. This method can be applied to a terminal, specifically executed by hardware or software within the terminal. The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).
[0062] Reference Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the text matching method based on probability distribution of this application.
[0063] In this embodiment, the text matching method based on probability distribution includes steps S10 to S40:
[0064] Step S10: Obtain the semantic feature distribution of each professional knowledge text from the professional knowledge base to obtain the knowledge probability distribution set.
[0065] It should be noted that professional knowledge text can be understood as specific text content stored in a professional knowledge base, such as professional documents or materials, covering structured knowledge in a specific field. Semantic feature distribution can be understood as representing text as a probability distribution through a probabilistic model (such as embedding space mapping) to capture its inherent semantic uncertainty. The knowledge probability distribution set can be a set composed of semantic feature distributions extracted from all professional knowledge texts in the professional knowledge base, used to represent the abstract features of knowledge.
[0066] Step S20: Obtain the user input text and calculate the corresponding user text probability distribution.
[0067] It should be noted that user input text, i.e., the query text provided by the user, may contain errors or noise. The user text probability distribution refers to the semantic feature distribution calculated by applying the same probability model to the user input text, representing the semantic intent of the query. For example, regardless of the method used, probability distributions can be calculated for both professional knowledge text and user input text. The user input text is processed in the same way as the professional knowledge text to obtain its probability distribution. ,in{ Consistent with the vocabulary set of professional knowledge, the sub-flowchart references... Figure 2 ,include:
[0068] (1) Obtaining user input: After the user completes the input of all text information, the system obtains the text content entered by the user in real time and prepares to submit it to the large language model for processing. After completion, proceed to step (2).
[0069] (2) Calculate word vectors U: Using the same BERT model as the one used for professional knowledge processing, calculate the word vectors U of the user input text. After completion, proceed to step (3).
[0070] (3) Calculate the probability distribution Based on the word vector U input by the user, the probability distribution of each dimension uj is calculated using the probability distribution formula, thus obtaining the probability distribution of the user input. .
[0071] Step S30: Calculate the similarity distance between the user text probability distribution and each distribution in the knowledge probability distribution set.
[0072] It should be noted that this refers to a distance metric used to quantify the difference between two probability distributions and to assess semantic similarity. Similarity measurement methods, such as Hellinger distance or Wasserstein distance, can be used to calculate the similarity distance between a user's text probability distribution and each distribution in the knowledge probability distribution set.
[0073] Step S40: Determine the text matching result based on the minimum similarity distance.
[0074] It should be noted that the best-matched professional knowledge text, determined by the minimum similarity distance, is used as the final output.
[0075] This embodiment employs a technique of extracting semantic feature distributions from a professional knowledge base, representing professional knowledge as a set of knowledge probability distributions. This addresses the problem of failing to capture semantic uncertainty due to the use of fixed word vectors. By abstracting the inherent features of text through probability distributions, a standardized representation of the knowledge base is achieved. Using the same method as calculating the probability distribution of user input text, the limitation of word vector distortion when user input is incorrect is overcome. By capturing the core intent of the query rather than surface vocabulary through a probability distribution model, the fault tolerance of text matching is significantly improved. The technique of calculating similarity distance addresses the sensitivity of vector similarity calculation to small changes. By measuring overall semantic similarity through distribution distance rather than local matching, the comparison process can tolerate distribution shifts. Finally, the method of determining the matching result based on the minimum similarity distance solves the problem of low matching accuracy in traditional techniques, achieving stable retrieval of semantically relevant professional knowledge even under high noise conditions, significantly improving fault tolerance and matching accuracy.
[0076] Reference Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the text matching method based on probability distribution of this application. Figure 1 The first embodiment shown presents a second embodiment of the text matching method based on probability distribution of this application.
[0077] In the second embodiment, step S10 includes:
[0078] Step S101: Obtain each professional knowledge text from the professional knowledge base and calculate the word frequency or word vector of the professional knowledge text.
[0079] It should be noted that when dealing with texts with a relatively significant vocabulary distribution, the word frequency of the specialized knowledge text can be calculated. When it is necessary to capture the deep semantic features of the text, the word vectors of the specialized knowledge text can be calculated. Word frequency can be understood as quantifying the text content by statistically analyzing the frequency of word occurrences in the text. Pre-trained models can be used to map words into dense vectors to capture semantic information. The pre-trained model can be a Bidirectional Encoder Representations from Transformers (BERT) model.
[0080] Step S102: Calculate the probability distribution of knowledge text based on word frequency or word vector to obtain a set of knowledge probability distributions.
[0081] It should be noted that the probability distribution of knowledge text can be a semantic feature distribution generated by using a probability calculation model based on word frequency or word vectors, which represents the potential semantic structure of knowledge text.
[0082] For example, when calculating the probability distribution based on the BERT model, each piece of professional knowledge text is transformed into a k-dimensional word vector. ,in( = [ , ,…, ,…, ]), ( k =768). Next, the word vectors are normalized into a probability distribution. Its calculation formula is expressed as formula (1).
[0083] (1)
[0084] When calculating the probability distribution based on word frequency, a set of specialized vocabulary within the field is compiled based on all standard documents. },in The set is indexed by the professional terminology and has a size of [size missing]. k Then, the word frequency in each professional knowledge text was statistically analyzed. That is, professional terminology Frequency of occurrence in the text. If a word does not appear, its frequency is set to 0. Probability distribution based on word frequency. The calculation formula is expressed as formula (2).
[0085] (2)
[0086] In this embodiment, the technique of calculating word frequencies or word vectors of professional knowledge text is employed, solving the problem that traditional techniques require precise input to generate effective vectors. Word frequencies or word vectors themselves can tolerate local lexical variations, providing a robust foundation for subsequent probability distribution calculations. The technique of generating probability distributions of knowledge text overcomes the limitation of traditional vector similarity being sensitive to input noise. Ultimately, a set of knowledge probability distributions is formed, transforming the professional knowledge base into a robust distributed representation, providing underlying support for fault-tolerant matching, and significantly improving the noise resistance of subsequent semantic similarity comparisons.
[0087] In one implementation, step S102 includes: calculating the probability distribution of knowledge texts based on word vectors, sequentially numbering each knowledge text in the professional knowledge base to obtain a knowledge number set, the knowledge number set including the mapping relationship between each knowledge text and each knowledge number; selecting a target knowledge text and vectorizing the target knowledge text to obtain word vectors; calculating the probability distribution of each dimension of the word vectors based on the probability distribution formula to obtain the probability distribution of the target knowledge text; establishing the mapping relationship between each knowledge number and the probability distribution of the target knowledge text to obtain a knowledge probability distribution set.
[0088] It should be noted that the knowledge ID set can be understood as an index that assigns unique numbers to all texts in the professional knowledge base in sequence, establishing a mapping relationship between texts and numbers. The target knowledge text can be understood as the professional knowledge text currently selected for probability distribution calculation, which can be dynamically specified through numbers. Vectorization processing can be understood as the process of transforming the target knowledge text into a dense vector sequence through a pre-trained model (such as BERT) to capture semantic features.
[0089] Specifically, the steps for selecting the target knowledge text include: if it is the first time selecting, then the knowledge text corresponding to the starting number in the knowledge number set is selected as the target knowledge text; if it is not the first time selecting, then the current number is incremented sequentially as the target number, and the knowledge text corresponding to the target number is selected as the target knowledge text.
[0090] It should be noted that the mechanism for controlling the traversal process through the numbering order—the first time it is processed, it starts from the initial number; subsequent times, the number is automatically incremented to achieve sequential traversal. Specifically, the steps to establish the mapping relationship between each knowledge number and the probability distribution of the target knowledge text, and to obtain the knowledge probability distribution set, include: traversing the knowledge number set; if there is unprocessed knowledge text, defining the unprocessed knowledge text as the target knowledge text; and returning to the step of vectorizing the target knowledge text to obtain word vectors.
[0091] It should be noted that each text is processed cyclically in numerical order to ensure full coverage of the knowledge base without any omissions.
[0092] For example, the specific process of using the BERT model to transform each knowledge point into a word vector and obtain a set of knowledge probability distributions is as follows: Figure 4 As shown. Specifically, it may include the following steps:
[0093] (1) Organize professional knowledge: Collect and organize professional knowledge in a certain field through manual means to ensure that the collected knowledge fully covers the field. After completion, proceed to step (2).
[0094] (2) Number the professional knowledge: Number all professional knowledge texts in a unified manner starting from 1, ensuring that each knowledge point has a unique number, and finally forming a set of professional knowledge numbers. After completion, proceed to step (3).
[0095] (3) Select knowledge point: If it is the first time to select, select the knowledge point numbered 1; if it is not the first time, select the knowledge point corresponding to the next number of the current knowledge point. After completion, proceed to step (4).
[0096] (4) Calculate word vectors Using the BERT model, the current knowledge points are transformed into word vectors. After completion, proceed to step (5).
[0097] (5) Calculate the probability distribution Based on the probability distribution formula, word vectors Each dimension Calculate its probability distribution. After completing the calculations for all dimensions, the probability distribution of this knowledge point is obtained. After completion, proceed to step (6).
[0098] (6) Have all knowledge points been traversed? Check whether all knowledge points have been processed and their probability distributions have been calculated. If there are still unprocessed knowledge points, return to step (3); otherwise, proceed to step (7).
[0099] (7) Form a set of probability distributions: Combine the probability distributions of all knowledge points into a set for later use.
[0100] This implementation employs a sequential numbering mapping technique for knowledge text (generating a set of knowledge numbers), solving the memory overflow problem caused by the traditional method's requirement to load the entire database at once. It achieves scalable processing of large-scale knowledge bases through step-by-step indexing. By dynamically selecting target knowledge texts and incrementally numbering them, it overcomes the limitations of inefficient traditional batch computation. Based on the vectorization and probability distribution calculation of the target text, it addresses the sensitivity of traditional direct word vector matching to local noise. By calculating the probability distribution of each dimension of the word vectors based on probability distribution formulas, it obtains the probability distribution of the target knowledge text, giving the representation statistical fault tolerance. Finally, by traversing and establishing the mapping relationship between numbers and distributions, compared to traditional static vector libraries, it achieves dynamic construction and structured storage of the knowledge base's probability distribution, providing a foundation for subsequent highly robust similarity comparisons and significantly improving the system's stability and noise resistance when processing massive amounts of knowledge.
[0101] In one implementation, step S102 further includes: calculating the probability distribution of knowledge texts based on word frequency, obtaining various professional knowledge texts from the professional knowledge base, extracting professional vocabulary from each professional knowledge text to obtain the word frequency of the professional vocabulary; based on the word frequency of the professional vocabulary, introducing inverse document frequency to calculate the weight of the professional vocabulary to obtain the weighted calculation result; and normalizing the weighted calculation result to obtain the knowledge probability distribution set.
[0102] For example, domain-specific words can be filtered from professional knowledge texts using domain dictionaries or term recognition models. The term frequency (TF) of a professional term refers to the frequency with which it appears in the current knowledge text. Inverse document frequency (IDF) is an indicator that quantifies the scarcity of words. Multiplying the term frequency (TF) by the inverse document frequency (IDF) yields the TF-IDF, which amplifies the discriminative weight of key domain terms. The weighted calculation result is then normalized, such as by scaling it to the [0, 1] interval (e.g., dividing by the vector magnitude), to obtain a set of knowledge probability distributions.
[0103] For example, the formula for calculating TF-IDF is expressed as formula (3).
[0104] (3)
[0105] In formula (3), Specialized terms Term frequency - inverse document frequency, Specialized terms word frequency, Specialized terms Inverse document frequency. and The calculation methods are expressed as formula (4) and formula (5) respectively.
[0106] (4)
[0107] (5)
[0108] In formulas (4) and (5), word frequency Specialized terms The frequency of occurrence in current knowledge texts Total number of documents For containing words The number of documents. After normalization, the probability distribution is finally obtained, and its calculation formula is expressed as formula (6).
[0109] (6)
[0110] This implementation employs a technique of extracting specialized vocabulary and calculating word frequencies to address the semantic ambiguity caused by the inclusion of generic words in traditional word frequency statistics. It strengthens the expression of core features by focusing on key domain terms. Furthermore, it introduces inverse document frequency weighted calculation to resolve the oversensitivity of simple word frequency to common terms. By suppressing high-frequency generic words and highlighting low-frequency specialized words, it improves the domain-specific discriminative power of the distribution representation. Finally, by normalizing and generating a probability distribution set, it transforms text semantics into a standardized probability distribution, ensuring that subsequent similarity distance calculations are not affected by text length differences, significantly improving the robustness and fault tolerance of semantic matching in specialized scenarios.
[0111] In one implementation, step S30 includes: calculating the similarity distance between the user text probability distribution and each distribution in the knowledge probability distribution set using the Bhattacharyya distance formula; wherein, the Bhattacharyya distance formula is expressed as formula (7).
[0112] (7)
[0113] In formula (7), Indicates the first The probability distribution of a knowledge text. This represents the probability distribution of the user's input text. Indicates the first The similarity distance between the probability distribution of a knowledge text and the probability distribution of the user input text.
[0114] In this implementation, the Bhattacharyya distance between the user's input text and all professional knowledge texts is calculated, and the professional knowledge text with the smallest distance is found from the calculation results. The professional knowledge corresponding to this text is semantically closest to the user's input text, and therefore can be considered the professional knowledge that best matches the user's input. The matched professional knowledge text is submitted together with the user's input text to a large language model, allowing the model to generate more accurate answers or classification results. This approach is suitable for tasks such as text classification, intelligent question answering, and information retrieval, and can significantly improve the robustness and accuracy of matching.
[0115] In one implementation, the sub-process of step S40 is as follows: Figure 5 As shown. Specifically, it may include the following steps:
[0116] (1) Initialize MinD and BestIndex: Define the variable MinD to store the minimum similarity distance and the variable BestIndex to store the corresponding professional knowledge index. The initial values are set as MinD = 1000 and BestIndex = -1. After completion, proceed to step (2).
[0117] (2) Select professional knowledge: Select a probability distribution of professional knowledge from the set of probability distributions of professional knowledge for processing. If it is the first selection, select the first record in the set; otherwise, select the next probability distribution of the current record. After completion, proceed to step (3).
[0118] (3) Obtain the probability distribution Pi(wj): Extract the probability distribution of the current professional knowledge and its corresponding serial number . After completion, proceed to step (4).
[0119] (4) Calculate the Bhattacharyya distance : Using the Bhattacharyya distance formula, calculate the similarity distance Di between the user input probability distribution Q(uj) and the current professional knowledge probability distribution Pi(wj). After completion, proceed to step (5).
[0120] (5) <MinD: Compare and the current minimum distance MinD. If <MinD, it indicates that the current distance is smaller, and proceed to step (6); otherwise, directly proceed to step (7).
[0121] (6) Smaller MinD and BestIndex: Update MinD to the current smaller distance , and at the same time update BestIndex to the serial number of the current professional knowledge . After completion, proceed to step (7).
[0122] (7) Whether the traversal is completed: Check whether the similarity distances of the probability distributions of all professional knowledge have been calculated. If so, proceed to step (8); otherwise, return to step (2) to continue processing the un-traversed professional knowledge.
[0123] (8) Determine the best match: Based on the professional knowledge corresponding to BestIndex, determine the best professional knowledge that matches the user input, and end the process.
[0124] In this embodiment, by calculating the similarity of the probability distributions between the user input and the professional knowledge, combined with the Bhattacharyya distance, the professional knowledge that best matches the user input is selected. Then, combined with the RAG technology, the user input and the matching professional knowledge are submitted to the large language model together, and the model performs reasoning on the user input and generates the most appropriate professional knowledge answer, significantly improving the robustness and accuracy of the matching.
[0125] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the text matching method based on probability distribution in this application. Any simple transformations based on this technical concept are within the protection scope of this application.
[0126] This application also provides a text matching device based on probability distribution, please refer to... Figure 6 The probability distribution-based text matching device includes:
[0127] The first probability calculation module 10 is used to obtain the semantic feature distribution of each professional knowledge text from the professional knowledge base and obtain a set of knowledge probability distributions.
[0128] The second probability calculation module 20 is used to acquire user input text and calculate the corresponding user text probability distribution.
[0129] The similarity distance calculation module 30 is used to calculate the similarity distance between the user text probability distribution and each distribution in the knowledge probability distribution set;
[0130] The text matching determination module 40 is used to determine the text matching result based on the minimum similarity distance. The probability distribution-based text matching device provided in this application, employing the probability distribution-based text matching method in the above embodiments, can solve the technical problem of low fault tolerance in text matching when user input information contains noise. Compared with the prior art, the beneficial effects of the probability distribution-based text matching device provided in this application are the same as those of the probability distribution-based text matching method provided in the above embodiments, and other technical features in the probability distribution-based text matching device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.
[0131] This application provides a probability distribution-based text matching device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the probability distribution-based text matching method in Embodiment 1 above.
[0132] The following is for reference. Figure 7The diagram illustrates a structural schematic of a probability distribution-based text matching device suitable for implementing embodiments of this application. The probability distribution-based text matching device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The text matching device based on probability distribution shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0133] like Figure 7 As shown, the probability distribution-based text matching device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the probability distribution-based text matching device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows probability-based text matching devices to communicate wirelessly or wiredly with other devices to exchange data. Although... Figure 7 Probability distribution-based text matching devices with various systems are shown; however, it should be understood that implementation or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.
[0134] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0135] The probability distribution-based text matching device provided in this application, employing the probability distribution-based text matching method in the above embodiments, can solve the technical problem of low fault tolerance in text matching when user input information contains noise. Compared with the prior art, the beneficial effects of the probability distribution-based text matching device provided in this application are the same as those of the probability distribution-based text matching method provided in the above embodiments, and other technical features in this probability distribution-based text matching device are the same as those disclosed in the previous embodiment method, and will not be repeated here.
[0136] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0137] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0138] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the probability distribution-based text matching method in the above embodiments.
[0139] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0140] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0141] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described probability distribution-based text matching method. This addresses the technical problem of low fault tolerance in text matching when user input information contains noise. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the probability distribution-based text matching method provided in the above embodiments, and will not be elaborated upon here.
[0142] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.
Claims
1. A text matching method based on probability distribution, characterized in that, The method includes: Obtain various professional knowledge texts from the professional knowledge base, and calculate the word frequency or word vector of the professional knowledge texts; The probability distribution of knowledge text is calculated based on word frequency or word vectors to obtain a set of knowledge probability distributions; Obtain the user input text and calculate the corresponding user text probability distribution; Calculate the similarity distance between the user text probability distribution and each distribution in the knowledge probability distribution set; The text matching result is determined based on the minimum similarity distance; The step of calculating the probability distribution of knowledge text based on word frequency or word vectors to obtain a set of knowledge probability distributions includes: If the probability distribution of knowledge texts is calculated based on word frequency, each knowledge text is obtained from the knowledge base, and the professional vocabulary of each knowledge text is extracted to obtain the word frequency of the professional vocabulary; Based on the word frequency of the aforementioned professional terms, inverse document frequency is introduced to calculate the weight of the professional terms, and a weighted calculation result is obtained; The weighted calculation results are normalized to obtain a set of knowledge probability distributions; If the probability distribution of knowledge texts is calculated based on word vectors, and each knowledge text in the professional knowledge base is numbered sequentially, a set of knowledge numbers is obtained. The set of knowledge numbers includes the mapping relationship between each knowledge text and each knowledge number. Select the target knowledge text and vectorize it to obtain word vectors; Based on the probability distribution formula, the probability distribution of the target knowledge text is calculated for each dimension of the word vector. Establish a mapping relationship between each knowledge number and the probability distribution of the target knowledge text to obtain a set of knowledge probability distributions.
2. The method as described in claim 1, characterized in that, The step of selecting the target knowledge text includes: If this is the first time selecting, the knowledge text corresponding to the starting number in the knowledge number set will be selected as the target knowledge text. If this is not the first selection, the current number will be incremented sequentially to become the target number, and the knowledge text corresponding to the target number will be selected as the target knowledge text.
3. The method as described in claim 1, characterized in that, The step of establishing the mapping relationship between each knowledge number and the probability distribution of the target knowledge text to obtain the knowledge probability distribution set includes: Traverse the knowledge ID set; if there is unprocessed knowledge text, define the unprocessed knowledge text as the target knowledge text. Return to the step of vectorizing the target knowledge text to obtain word vectors.
4. The method as described in claim 1, characterized in that, The step of calculating the similarity distance between the user text probability distribution and each distribution in the knowledge probability distribution set includes: The similarity distance between the user text probability distribution and each distribution in the knowledge probability distribution set is calculated using the Bach distance formula. The Bach distance formula is expressed as follows: ； in, This indicates the sequence number of the specialized vocabulary in the vocabulary set. This indicates the number of specialized terms in the vocabulary set. Indicates the first The probability distribution of a knowledge text. Represents word vectors in knowledge text. Indicates technical terms, This represents the probability distribution of the user's input text. Word vectors representing user-input text. This indicates that the user has entered text. Indicates the first The similarity distance between the probability distribution of a knowledge text and the probability distribution of the user input text.
5. A text matching device based on probability distribution, characterized in that, The device includes: The first probability calculation module is used to obtain various professional knowledge texts from the professional knowledge base and calculate the word frequency or word vector of the professional knowledge text; calculate the probability distribution of the knowledge text based on the word frequency or word vector to obtain a set of knowledge probability distributions; The second probability calculation module is used to obtain user input text and calculate the corresponding user text probability distribution. The similarity distance calculation module is used to calculate the similarity distance between the user text probability distribution and each distribution in the knowledge probability distribution set; The text matching determination module is used to determine the text matching result based on the minimum similarity distance. The first probability calculation module is further configured to: calculate the probability distribution of knowledge texts based on word frequency; obtain various professional knowledge texts from the professional knowledge base; extract the professional vocabulary of each professional knowledge text to obtain the word frequency of the professional vocabulary; calculate the weight of the professional vocabulary based on the word frequency of the professional vocabulary by introducing inverse document frequency to obtain a weighted calculation result; normalize the weighted calculation result to obtain a set of knowledge probability distributions; calculate the probability distribution of knowledge texts based on word vectors; number each knowledge text in the professional knowledge base in sequence to obtain a set of knowledge numbers, the set of knowledge numbers including the mapping relationship between each knowledge text and each knowledge number; select a target knowledge text and vectorize the target knowledge text to obtain word vectors; calculate the probability distribution of each dimension of the word vectors based on the probability distribution formula to obtain the probability distribution of the target knowledge text; establish the mapping relationship between each knowledge number and the probability distribution of the target knowledge text to obtain a set of knowledge probability distributions.
6. A text matching device based on probability distribution, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the probability distribution-based text matching method as described in any one of claims 1 to 4.
7. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the text matching method based on probability distribution as described in any one of claims 1 to 4.