Semantic relationship recognition model training method and device and semantic relationship recognition method and device
By constructing recognition prompt text and knowledge distillation technology, the inference ability of large language models is transferred to small models, solving the high computational cost problem of large language models on resource-limited equipment, and achieving efficient and accurate semantic relationship recognition.
Patent Information
- Application Number
- CN202510028804.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-08
- Publication Date
- 2025-03-28
AI Technical Summary
When using large language models for semantic association recognition, the inference process requires high computational costs, especially on devices with limited resources.
By constructing the recognition prompt text, the teacher model is guided to identify the semantic relationship of each text in the teacher training text according to the thinking chain, and through knowledge distillation technology, the reasoning ability of the teacher model is distilled onto the student model with fewer parameters.
It realizes the high accuracy and reasoning ability of the model while reducing the computational cost, so that the student model can still have effective inference performance while constrained computing resources and save computing costs.
Smart Images
Figure CN119721051A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to a semantic relationship recognition model training, a semantic relationship recognition method and a device. Background Art
[0002] Semantic relationship recognition is to analyze the semantic connections between words, phrases or sentences, use these connections for logical reasoning, and thus understand, explain or infer deeper knowledge or information to identify the semantic associations between different knowledge points.
[0003] Since large language models (hereinafter referred to as "large models") have powerful knowledge reasoning capabilities, they are currently mostly used to identify and reason about semantic associations between knowledge points. However, when using large models for semantic association identification, reasoning often requires high computing costs, especially in actual application scenarios, where devices with limited resources cannot afford the high computing costs. Summary of the invention
[0004] The present invention provides a semantic relationship recognition model training, a semantic relationship recognition method and a device, so as to solve the defects existing in the prior art.
[0005] The present invention provides a semantic relationship recognition model training method, comprising the following steps: Based on the teacher training text, the semantic relationship labels between the texts in the teacher training text, and the recognition prompt text, a teacher model is trained, wherein the recognition prompt text is used to guide the teacher model to perform semantic relationship recognition on the texts in the teacher training text according to the thinking chain; Based on the teacher model, semantic relationship recognition is performed on each text in the student training text to obtain a teacher recognition result; Based on the student model, semantic relationship recognition is performed on each text in the student training text to obtain a student recognition result, wherein the number of parameters of the student model is less than the number of parameters of the teacher model; The student model is trained with the training goal of minimizing the difference between the teacher's recognition result and the student's recognition result to obtain a semantic relationship recognition model.
[0006] According to a semantic relationship recognition model training method provided by the present invention, the teacher training text is determined based on the following steps: Determine a plurality of candidate description texts related to a target knowledge point, wherein the target knowledge point is a knowledge point corresponding to each text in the teacher training text; Based on the semantic similarity between each candidate description text and the corresponding target knowledge point, the teacher training text is determined from the multiple candidate description texts.
[0007] According to a semantic relationship recognition model training method provided by the present invention, the multiple candidate description texts are determined based on the following steps: Based on the target knowledge point, a search is performed in the knowledge base to obtain a plurality of knowledge triples matching the target knowledge point; Based on a large language model, the semantic information of each knowledge triple is applied to perform retrieval enhancement on each knowledge triple, and a knowledge description text is generated to describe each knowledge triple; Based on the large language model, the knowledge description text of each knowledge triple is subjected to thought chain distillation prompts to obtain corresponding candidate description texts, and each candidate description text is used to describe the semantic relationship between two knowledge points in the corresponding knowledge triple in a thought chain manner.
[0008] According to a semantic relationship recognition model training method provided by the present invention, the method of determining the teacher training text from the plurality of candidate description texts based on the semantic similarity between each candidate description text and the corresponding target knowledge point comprises: Determining an initial training text from the plurality of candidate description texts based on the semantic similarity between each candidate description text and the corresponding target knowledge point; The teacher training text is determined from each initial training text based on the occurrence frequency and / or word frequency of the target keyword in each initial training text, and the target keyword is obtained by keyword extraction of each target knowledge point.
[0009] According to a semantic relationship recognition model training method provided by the present invention, the student model is trained with minimizing the difference between the teacher recognition result and the student recognition result as the training goal to obtain the semantic relationship recognition model, including: The student model is trained with the training objectives of minimizing the difference between the teacher recognition result and the student recognition result, and minimizing the difference between the semantic relationship labels between the student recognition result and each text in the student training text, to obtain the semantic relationship recognition model.
[0010] According to a semantic relationship recognition model training method provided by the present invention, the teacher model is trained based on a teacher training text, semantic relationship labels between texts in the teacher training text, and recognition prompt text, including: Based on the initial model of the teacher model, the teacher training text and the recognition prompt text are applied to determine the teacher prediction result; Determining a training loss value based on a difference between the teacher's prediction result and the semantic relationship labels between the texts in the teacher's training text; Based on the training loss value, the parameters of the initial model are fine-tuned to obtain the teacher model.
[0011] The present invention also provides a semantic relationship recognition method, comprising the following steps: determining a first target text and a second target text; Based on the semantic relationship recognition model, performing semantic relationship recognition on the first target text and the second target text to obtain a semantic relationship recognition result; The semantic relationship recognition model is trained based on the semantic relationship recognition model training method as described in any one of the above items.
[0012] The present invention also provides a semantic relationship recognition model training device, comprising the following modules: A first training unit is used to train a teacher model based on a teacher training text, semantic relationship labels between texts in the teacher training text, and recognition prompt text, wherein the recognition prompt text is used to guide the teacher model to perform semantic relationship recognition on each text in the teacher training text according to a thinking chain; A teacher identification unit, used to identify the semantic relationship of each text in the student training text based on the teacher model to obtain a teacher identification result; A student identification unit, used for performing semantic relationship identification on each text in the student training text based on a student model to obtain a student identification result, wherein the parameter amount of the student model is less than the parameter amount of the teacher model; The second training unit is used to train the student model with minimizing the difference between the teacher's recognition result and the student's recognition result as the training goal, so as to obtain a semantic relationship recognition model.
[0013] The present invention also provides a semantic relationship recognition device, comprising the following modules: A text determination unit, used for determining a first target text and a second target text; A relationship recognition unit, configured to perform semantic relationship recognition on the first target text and the second target text based on a semantic relationship recognition model to obtain a semantic relationship recognition result; The semantic relationship recognition model is trained based on the semantic relationship recognition model training method as described in any one of the above items.
[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any of the above-described semantic relationship recognition model training methods or semantic relationship recognition methods.
[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the semantic relationship recognition model training methods or semantic relationship recognition methods described above.
[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned semantic relationship recognition model training methods or semantic relationship recognition methods.
[0017] The semantic relationship recognition model training, semantic relationship recognition method and device provided by the present invention, by constructing recognition prompt text, guide the teacher model to perform semantic relationship recognition on each text in the teacher training text according to the thinking chain, thereby realizing the decomposition of a large task into a series of logically interrelated steps, and helping the teacher model to gradually derive a conclusion. Through step-by-step derivation, the teacher model can automatically generate a reasonable reasoning process according to human thinking, and give a final judgment based on the thinking chain, so that the entire reasoning process does not only rely on simple semantic matching, but through logical reasoning, the complex knowledge point relationship is gradually decomposed to obtain the final conclusion and ensure the reasoning accuracy. In addition, the present invention takes minimizing the difference between the teacher recognition result and the student recognition result as the training goal, and trains the student model, which not only enables the student model to gradually approach the performance of the teacher model and accurately identify semantic relationships, but also can maintain a smaller model scale and faster reasoning speed of the student model, saving computing costs. Furthermore, the parameter scale of the student model is smaller than that of the teacher model, so the student model is usually regarded as a small model, while the teacher model is regarded as a large model. Through the knowledge transfer mechanism, the rich knowledge learned by the large model is effectively transferred to the small model, which not only ensures that the small model maintains a high accuracy in reasoning tasks, but also significantly reduces the computing resources and storage costs required for training. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 It is a flowchart of the semantic relationship recognition model training method provided by the present invention.
[0020] Figure 2 It is a flow chart of another semantic relationship recognition model training method provided by the present invention.
[0021] Figure 3It is a flow chart of the semantic relationship recognition method provided by the present invention.
[0022] Figure 4 It is a structural schematic diagram of the semantic relationship recognition model training device provided by the present invention.
[0023] Figure 5 It is a structural schematic diagram of the semantic relationship recognition device provided by the present invention.
[0024] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] Since large models have powerful knowledge reasoning capabilities, they are currently used to identify and reason about semantic associations between knowledge points. However, when using large models for semantic association identification, reasoning often requires high computational costs, especially in practical application scenarios, where devices with limited resources cannot afford the high computational overhead. The computational requirements of large models limit their widespread use in practical applications, especially in tasks that require fast response or low computational costs.
[0027] Therefore, the present invention uses knowledge distillation technology to distill the reasoning ability of the teacher model with a large number of parameters to the student model with a smaller number of parameters, ensuring that the high accuracy and reasoning ability of the model are maintained while the reasoning cost is greatly reduced, so that the student model can still have effective reasoning performance when computing resources are limited.
[0028] in, Figure 1 is a flow chart of the semantic relationship recognition model training method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .
[0029] Step 110, based on the teacher training text, the semantic relationship labels between the texts in the teacher training text and the recognition prompt text, a teacher model is trained. The recognition prompt text is used to guide the teacher model to perform semantic relationship recognition on the texts in the teacher training text according to the thinking chain.
[0030] Here, the teacher training text refers to a text set used to train the teacher model. The teacher training text may include multiple texts, and each text corresponds to a corresponding knowledge point. The semantic relationship labels between the texts in the teacher training text can be understood as the association between the knowledge points in each text. The relationship between the knowledge points may be of different types such as subordination and dependency. For example, the knowledge point of text A is relativity, and the knowledge point of text B is general relativity. The semantic relationship label between text A and text B can be a subordination relationship; the knowledge point of text C is gradient descent, and the knowledge point of text D is back propagation. Only after understanding gradient descent can we understand back propagation, that is, the knowledge point in text D depends on the knowledge point in text C, then the semantic relationship label between text C and text D can be a dependency relationship.
[0031] In addition, considering that the task of identifying the semantic relationship of knowledge points generally encounters the challenge of scarcity of high-quality annotated data, which is particularly prominent in professional fields, it is particularly difficult to obtain such annotated data. In this regard, the embodiment of the present invention constructs a recognition prompt text to guide the teacher model to perform semantic relationship recognition on each text in the teacher training text according to the chain of thought. Among them, the chain of thought can be understood as a series of logically interrelated and sequentially arranged reasoning steps or links, so as to decompose a large task into a series of logically interrelated steps, thereby helping the teacher model to gradually derive conclusions according to human thinking. The recognition prompt text can be a Prompt text.
[0032] For example, the recognition prompt text may be "Text 1 corresponds to knowledge point A, text 2 corresponds to knowledge point 2, first determine whether knowledge point A contains knowledge point B. If not, check whether understanding knowledge point B depends on understanding knowledge point A. Finally, provide an explanation of the relationship between the two."
[0033] It can be seen that the above recognition prompt text guides the teacher model to first consider the subordinate relationship between knowledge point A and knowledge point B, and then gradually infer whether they have a dependent relationship. Through step-by-step deduction, the teacher model can automatically generate a reasonable reasoning process and give a final judgment based on the thinking chain, so that the entire reasoning process does not only rely on simple semantic matching, but through logical reasoning, the complex relationship between knowledge points is gradually decomposed to draw the final conclusion and ensure the reasoning accuracy.
[0034] Exemplarily, the teacher training text and the recognition prompt text can be input into the initial teacher model, and the initial teacher model performs semantic relationship recognition on each text in the teacher training text according to the thought chain in the recognition prompt text to obtain a semantic recognition result. Based on the difference between the semantic recognition result and the semantic relationship labels between each text in the teacher training text, the initial teacher model is trained to obtain a teacher model.
[0035] Step 120: Based on the teacher model, semantic relationship recognition is performed on each text in the student training text to obtain a teacher recognition result.
[0036] Step 130: Based on the student model, semantic relationship recognition is performed on each text in the student training text to obtain a student recognition result. The number of parameters of the student model is less than that of the teacher model.
[0037] Step 140: Taking minimizing the difference between the teacher's recognition result and the student's recognition result as the training goal, the student model is trained to obtain a semantic relationship recognition model.
[0038] Considering that the teacher model has powerful reasoning ability, its reasoning process usually consumes a lot of computing resources due to its large scale. The embodiment of the present invention allows the student model to learn the output of the teacher model, thereby inheriting the reasoning ability of the teacher model and significantly reducing the computational overhead required for reasoning. During the training process, the smaller-scale student model not only learns the teacher recognition results of the teacher model, but also learns the feature representation and probability distribution of the teacher model in the middle layer, thereby effectively improving the reasoning accuracy of the student model. In addition, under the framework of model compression and knowledge distillation, the parameter scale of the student model is smaller than that of the teacher model, so the student model is usually regarded as a lightweight model, while the teacher model is regarded as a large model. Based on this setting, through the knowledge transfer mechanism, the rich knowledge learned by the large model is effectively transferred to the lightweight model, which can not only ensure that the lightweight model maintains a high accuracy in the reasoning task, but also significantly reduce the computing resources and storage costs required for training.
[0039] Specifically, the student training text refers to a text set used to train the student model, and the student training text may include multiple texts, each of which corresponds to a corresponding knowledge point. The student training text may be completely identical to the teacher training text, may be partially identical to the teacher training text, or may be completely different from the teacher training text.
[0040] After the teacher model is trained, it has the ability to accurately identify the semantic relationship between texts. Therefore, after the student training text is input into the teacher model, the teacher model can accurately identify the semantic relationship between each text in the student training text and obtain the teacher recognition result. In addition, the student training text is input into the student model, and the student model predicts the semantic relationship between each text in the student training text to obtain the student recognition result.
[0041] The teacher's recognition results are used as soft labels, and the student model is trained with the goal of minimizing the difference between the teacher's recognition results and the student's recognition results, which can guide the student model to learn more refined and accurate semantic relationship features. The above soft labels not only contain the direct semantic relationship between texts, but also imply the contextual information, semantic hierarchy and logical relationship captured by the teacher model during the reasoning process, which are difficult for the student model to fully capture when trained independently.
[0042] By training with the teacher's recognition results as soft labels, the student model can gradually approach the performance of the teacher model while maintaining its smaller model size and faster inference speed, which helps the student model better generalize to unseen data and improve its accuracy and robustness in various semantic relationship recognition tasks.
[0043] Exemplarily, a KL divergence (Kullback-Leibler divergence) may be determined based on the difference between the teacher recognition result and the student recognition result, and the parameters of the student model may be updated based on the KL divergence to obtain a semantic relationship recognition model.
[0044] The semantic relationship recognition model training method provided by the embodiment of the present invention guides the teacher model to perform semantic relationship recognition on each text in the teacher training text according to the chain of thinking by constructing recognition prompt text, thereby realizing the decomposition of large tasks into a series of logically interrelated steps, and helping the teacher model to gradually derive conclusions. Through step-by-step derivation, the teacher model can automatically generate a reasonable reasoning process according to human thinking, and give a final judgment according to the chain of thinking, so that the entire reasoning process does not only rely on simple semantic matching, but through logical reasoning, the complex knowledge point relationship is gradually decomposed to obtain the final conclusion and ensure the reasoning accuracy. In addition, the embodiment of the present invention takes minimizing the difference between the teacher recognition result and the student recognition result as the training goal, and trains the student model, which not only enables the student model to gradually approach the performance of the teacher model and accurately recognize the semantic relationship, but also can maintain the smaller model scale and faster reasoning speed of the student model, saving computing costs. Furthermore, the parameter scale of the student model is smaller than the parameter scale of the teacher model, so the student model is usually regarded as a lightweight model (i.e., a small model), while the teacher model is regarded as a large model (i.e., a large model). Based on this setting, the rich knowledge learned by the large model can be effectively transferred to the small model through the knowledge transfer mechanism, which not only ensures that the small model maintains a high accuracy in reasoning tasks, but also significantly reduces the computing resources and storage costs required for training.
[0045] Based on the above embodiment, the teacher training text is determined based on the following steps: Determine multiple candidate description texts related to target knowledge points, where the target knowledge points are knowledge points corresponding to the texts in the teacher training texts; Based on the semantic similarity between each candidate description text and the corresponding target knowledge point, a teacher training text is determined from multiple candidate description texts.
[0046] Considering that when using large models for semantic relationship recognition, the training sets of large models usually have problems such as data scarcity, incomplete annotations, and unclear semantic levels, the large models show instability and errors when processing the relationship between knowledge points. This problem is particularly evident in professional fields. Proprietary knowledge in the field is difficult to obtain through general data sets, and the difficulty of data annotation is also greatly increased.
[0047] To this end, an embodiment of the present invention accurately extracts descriptive information related to the target knowledge points from large-scale structured and unstructured data sources, and further filters out teacher training text from these descriptive information, thereby effectively filtering out low-relevant content and ensuring that the teacher training text can accurately describe the corresponding target knowledge points, providing high-quality basic data for semantic relationship recognition and reasoning, thereby improving the accuracy and reliability of subsequent reasoning.
[0048] The target knowledge point is the knowledge point corresponding to each text in the teacher training text, wherein each text may correspond to a different knowledge point. The candidate description text refers to the description text related to the target knowledge point, and each candidate description text may explain the target knowledge point in different ways, angles or details.
[0049] In addition, the semantic similarity between each candidate description text and the corresponding target knowledge point is used to characterize the relevance between each candidate description text and the target knowledge point. The higher the similarity, the higher the relevance between the corresponding candidate description text and the target knowledge point, and thus the candidate description text can accurately describe the target knowledge point, that is, the candidate description text can be regarded as high-quality training data, that is, the candidate description text can be used as a teacher training text. Similarly, the lower the similarity, the lower the relevance between the corresponding candidate description text and the target knowledge point, and thus the candidate description text can be filtered out as a low-relevance text.
[0050] Exemplarily, the text features of each candidate description text and the text features of the target knowledge point can be extracted through a pre-trained language model (such as BERT, RoBERTa, etc.), and the semantic similarity between each candidate description text and the target knowledge point is determined based on the similarity between the text features of each candidate description text and the text features of the target knowledge point. The distance between the text features of each candidate description text and the text features of the target knowledge point, cosine similarity, etc. can be used to characterize the semantic similarity between each candidate description text and the target knowledge point.
[0051] Among them, the formula for calculating cosine similarity is as follows: in, Represents the text feature vector of each candidate description text, The text feature vector representing the target knowledge point, is the dot product of the vectors, and are the moduli of the two vectors.
[0052] It can be seen that the embodiment of the present invention can determine the text most relevant to the target knowledge point from multiple candidate description texts as the teacher training text based on the semantic similarity between each candidate description text and the corresponding target knowledge point, thereby providing high-quality training data for the semantic relationship recognition model, improving the coverage and accuracy of the training data, so that the model can still ensure the accuracy and reliability of reasoning in the absence of labeled data.
[0053] Based on any of the above embodiments, multiple candidate description texts are determined based on the following steps: Based on the target knowledge point, the knowledge base is searched to obtain multiple knowledge triples matching the target knowledge point; Based on a large language model, the semantic information of each knowledge triple is applied to perform retrieval enhancement on each knowledge triple, and a knowledge description text is generated to describe each knowledge triple; Based on the large language model, the knowledge description text of each knowledge triple is subjected to thought chain distillation prompts to obtain the corresponding candidate description text. Each candidate description text is used to describe the semantic relationship between two knowledge points in the corresponding knowledge triple in a thought chain manner.
[0054] Specifically, a knowledge triple usually includes two entities and the relationship between the two entities, wherein in the knowledge triple that matches the target knowledge point, one entity corresponds to the target knowledge point, and the other entity corresponds to another knowledge point that has a relationship with the target knowledge point. In other words, the knowledge triple that matches the target knowledge point is used to provide various information related to the target knowledge point.
[0055] In addition, the knowledge description text is a natural language description of the knowledge triples, that is, the entities and relationships in the knowledge triples are converted into easy-to-understand text forms to obtain the knowledge description text. Among them, the knowledge description text is based on a large language model, and the semantic information of each knowledge triple is applied to perform retrieval enhancement on each knowledge triple. That is, the large language model uses an external knowledge base or information retrieval system to find information related to the knowledge triples. This information may include related text, context, definition, examples, etc., which are used to enhance the model's understanding and description capabilities of the triples. Through retrieval enhancement, the model can generate more accurate and comprehensive knowledge description texts.
[0056] After obtaining the knowledge description text, based on the large language model, the knowledge description text of each knowledge triple is subjected to thought chain distillation prompts to obtain the corresponding candidate description text. Among them, the thought distillation prompt is to further process and analyze the knowledge description text, extract the key information and logical structure therein, and present it in the form of a thought chain, which shows the reasoning steps and connections from one knowledge point to another. In an embodiment of the present invention, the purpose of the thought distillation prompt is to obtain the corresponding candidate description text, so that the candidate description text can describe the semantic relationship between the two knowledge points in the knowledge triple in a clearer and more coherent way, thereby reducing the understanding threshold during subsequent model training. Among them, the large language model in the embodiment of the present invention can be a teacher mode or other large language model, and the embodiment of the present invention does not make specific limitations on this.
[0057] For example, if knowledge point A in the knowledge triple is "BM algorithm", knowledge point B is "pattern string", and the relationship between knowledge point A and knowledge point B is a dependency relationship, then the generated knowledge description text can be "BM algorithm: BM algorithm, also known as the precise character algorithm..., pattern string: pattern string is a string to be matched, and the main string is a string containing the pattern string. In string processing, pattern string matching is a common operation...". After the knowledge description text is thought distilled, the candidate description text obtained is "To understand the BM algorithm, character matching and movement rules must be determined through pattern strings. If the concept and function of pattern strings are not understood, the matching and movement strategies based on pattern strings in the BM algorithm cannot be correctly understood. Therefore, only by understanding the pattern strings can the BM algorithm be understood."
[0058] Based on any of the above embodiments, determining a teacher training text from a plurality of candidate description texts based on the semantic similarity between each candidate description text and the corresponding target knowledge point includes: Based on the semantic similarity between each candidate description text and the corresponding target knowledge point, determining an initial training text from multiple candidate description texts; Based on the occurrence frequency and / or word frequency of the target keyword in each initial training text, the teacher training text is determined from each initial training text, and the target keyword is obtained by keyword extraction of each target knowledge point.
[0059] Considering that determining the initial training text based on semantic similarity can ensure a high relevance between the selected text and the target knowledge point in terms of content, the semantic similarity mainly reflects the degree of matching of the overall meaning and cannot accurately reflect the specific details of the target knowledge point. In this regard, after determining the initial training text from multiple candidate description texts based on the semantic similarity between each candidate description text and the corresponding target knowledge point, the embodiment of the present invention further performs screening based on the frequency of occurrence and / or word frequency of the target keyword in the initial training text, thereby being able to more accurately screen the teacher training text with a high relevance to the target knowledge point from the initial training text.
[0060] Among them, the target keywords can be understood as the keywords covered by the target knowledge points. Usually, the target knowledge points correspond to knowledge point description sentences. Keywords can be extracted from the knowledge point description sentences to determine the target keywords.
[0061] The frequency of occurrence of the target keyword in each initial training text is used to measure the ratio of the number of occurrences of the target keyword in each initial training text to the total length of each initial training text. The higher the frequency of occurrence, the more times the target keyword appears in the corresponding initial training text, and thus the higher the correlation between the corresponding initial training text and the target knowledge point.
[0062] The word frequency of the target keyword in each initial training text is used to measure the ratio of the number of times the target keyword appears in each initial training text to the total number of words in each initial training text. The higher the word frequency, the more frequently the target keyword appears in each initial training text, and the higher the correlation between the corresponding initial training text and the target knowledge point.
[0063] Based on this, the embodiment of the present invention can more accurately screen teacher training texts with a high relevance to the target knowledge points from the initial training texts based on the occurrence frequency and / or word frequency of the target keywords in each initial training text.
[0064] Optionally, the frequency of occurrence of the target keyword in each initial training text may be determined based on the following formula: in, Indicates target keywords In the initial training text The frequency of occurrence in is the total number of initial training texts, Contains the target keyword The number of initial training texts.
[0065] Optionally, the frequency of the target keyword in each initial training text may be determined based on the following formula: in, Is the target keyword In the initial training text The word frequency in represents the length of the initial training text, is the average length of all initial training texts, and is a tuning parameter.
[0066] Based on any of the above embodiments, the student model is trained with minimizing the difference between the teacher's recognition result and the student's recognition result as the training goal to obtain a semantic relationship recognition model, including: The student model is trained with the training objectives of minimizing the difference between the teacher's recognition results and the student's recognition results, and minimizing the difference between the semantic relationship labels between the student's recognition results and each text in the student's training text, to obtain a semantic relationship recognition model.
[0067] Considering that the recognition results of the teacher model can be used as the learning target of the student model, the teacher model itself may have problems such as overfitting, bias or limitations. If the accuracy of the student model imitating the teacher model is used as the training goal, the student model may inherit these problems, resulting in limited generalization ability for unseen texts. In addition, the recognition results of the teacher model may not be completely consistent with the real semantic relationship labels, because the real semantic relationships are often more complex and changeable. Therefore, in order to make the student model have stronger generalization ability and accuracy, the embodiment of the present invention takes minimizing the difference between the teacher recognition results and the student recognition results, and minimizing the difference between the semantic relationship labels between the student recognition results and the texts in the student training text as the training goals, and trains the student model to ensure that the trained semantic relationship recognition model is more accurate and reliable in understanding the semantic relationships in the real world.
[0068] Among them, minimizing the difference between the teacher's recognition results and the student's recognition results is used to train the student model's imitation ability, that is, it is used to train the student model to imitate the teacher model's recognition results as accurately as possible, so as to learn the teacher model's semantic relationship recognition ability.
[0069] Minimizing the difference between the student recognition results and the semantic relationship labels between the texts in the student training texts trains the generalization or understanding ability of the student model. The student model must not only be able to accurately recognize the semantic relationships in the student training texts, but also ensure that the student recognition results are consistent with the semantic relationship labels (i.e., the true labels) between the texts, thereby enhancing the generalization ability of the student model to unseen texts.
[0070] Exemplarily, the distillation loss value can be determined based on the difference between the teacher's recognition result and the student's recognition result, the recognition loss value can be determined based on the difference between the semantic relationship labels between the student's recognition result and each text in the student's training text, the training loss value can be determined based on the distillation loss value and the recognition loss value, and the student model can be trained based on the training loss value to obtain a semantic relationship recognition model.
[0071] Among them, the training loss value can be determined based on the following formula: in, represents the training loss value, represents the recognition loss value (which can be the output of the student model and the true label The cross entropy loss between represents the distillation loss value (which can be the output of the teacher model With the student model output KL divergence loss between represents the temperature parameter, which is used to smooth the probability distribution of the output, is the weight coefficient used to balance the two parts of loss.
[0072] Based on any of the above embodiments, a teacher model is trained based on the teacher training text, the semantic relationship labels between the texts in the teacher training text, and the recognition prompt text, including: Based on the initial model of the teacher model, the teacher training text and the recognition prompt text are applied to determine the teacher prediction results; Determine a training loss value based on the difference between the teacher's prediction results and the semantic relationship labels between the texts in the teacher's training texts; Based on the training loss value, the parameters of the initial model are fine-tuned to obtain the teacher model.
[0073] Specifically, in the semantic relationship recognition task, although a large-scale teacher model can accurately infer the teacher's prediction results, the huge number of parameters of the teacher model leads to high computational costs for training and reasoning. In order to reduce the computational cost while maintaining high reasoning performance, the embodiment of the present invention fine-tunes the parameters of the initial model based on the training loss value to obtain the teacher model.
[0074] Among them, the parameters of the initial model can be fine-tuned by the LoRA method. LoRA introduces a low-rank matrix into the weight matrix of the model, so that the model can quickly learn the features related to the new task without changing the main parameters of the model. The core idea of LoRA is to introduce two low-rank matrices into the weight matrix. and , the weight is adjusted by the following formula: In addition, the parameters of the initial model can be fine-tuned through the Adapter method. The Adapter inserts an additional lightweight network into the model to specifically handle specific tasks, which is expressed as: The embodiments of the present invention fine-tune the parameters of the initial model, thereby significantly reducing the cost of retraining the model on new tasks, the time for inference, and the consumption of computing resources without changing large-scale parameters, so that the model can maintain good performance in a wider range of application scenarios.
[0075] Based on any of the above embodiments, Figure 2 is a flow chart of another semantic relationship recognition model training method provided by the present invention, such as Figure 2 As shown, the method includes: Based on the target knowledge point, knowledge retrieval is performed in the third-party knowledge base to extract knowledge triples, which include knowledge point A, knowledge point B, and the relationship between knowledge points A and B. The target knowledge point can be knowledge point A or knowledge point B. For example, knowledge point A is the BM algorithm, knowledge point B is the pattern string, and the relationship between the two is a dependency relationship.
[0076] Next, based on a large model (such as a large model with a model parameter amount of 70B), the semantic information of the knowledge triples is applied to perform retrieval enhancement on the knowledge triples, and a knowledge description text for describing the knowledge triples is generated.
[0077] After obtaining the knowledge description text, based on the big model, the knowledge description text of each knowledge triple is subjected to thought chain distillation prompts to obtain the corresponding candidate description text. Each candidate description text is used to describe the semantic relationship between two knowledge points in the corresponding knowledge triple in a thought chain manner.
[0078] Then, a teacher training text is screened out from each candidate description text, wherein the screening process can be based on the semantic similarity between each candidate description text and the target knowledge point, determining the initial training text from multiple candidate description texts, and based on the appearance frequency and / or word frequency of the target keywords in the target knowledge point in each initial training text, screening the teacher training text from the initial training text.
[0079] Based on the teacher training text, the semantic relationship labels between the texts in the teacher training text, and the recognition prompt text, the initial model of the teacher model is fine-tuned to obtain the teacher model. After the teacher model is trained, the student training text is input into the teacher model to obtain the teacher recognition result, and the student training text is input into the student model to obtain the student recognition result. The student model is trained to obtain the semantic relationship recognition model with the training goal of minimizing the difference between the teacher recognition result and the student recognition result, and minimizing the difference between the student recognition result and the semantic relationship labels between the texts in the student training text. The teacher model can be a large model with a parameter amount of 70B, and the student model can be a small model with a parameter amount of 13B.
[0080] Based on any of the above embodiments, Figure 3 is a flow chart of the semantic relationship recognition method provided by the present invention, such as Figure 3 As shown, the method includes: Step 310: Determine a first target text and a second target text; Step 320: Based on the semantic relationship recognition model, semantic relationship recognition is performed on the first target text and the second target text to obtain a semantic relationship recognition result; The semantic relationship recognition model is trained based on the semantic relationship recognition model training method described in any of the above embodiments.
[0081] Specifically, the first target text and the second target text are texts whose semantic relationship needs to be determined. The first target text and the second target text can be directly input by the user, or can be obtained after voice recognition of the user's voice data. The embodiment of the present invention does not make specific limitations on this.
[0082] Since the semantic relationship recognition model described in any of the above embodiments not only accurately recognizes semantic relationships, but also can maintain a smaller model size and faster reasoning speed of the student model, saving computing costs, the semantic relationship recognition model can accurately and quickly obtain semantic relationship recognition results at a lower computing cost.
[0083] The semantic relationship recognition model training device provided by the present invention is described below. The semantic relationship recognition model training device described below and the semantic relationship recognition model training method described above can be referenced to each other.
[0084] Based on any of the above embodiments, Figure 4 is a structural diagram of the semantic relationship recognition model training device provided by the present invention, such as Figure 4 As shown, the device comprises: A first training unit 410 is used to train a teacher model based on a teacher training text, semantic relationship labels between texts in the teacher training text, and recognition prompt texts, wherein the recognition prompt texts are used to guide the teacher model to perform semantic relationship recognition on texts in the teacher training text according to a thinking chain; The teacher identification unit 420 is used to identify the semantic relationship of each text in the student training text based on the teacher model to obtain a teacher identification result; The student identification unit 430 is used to identify the semantic relationship of each text in the student training text based on the student model to obtain a student identification result, wherein the parameter amount of the student model is less than the parameter amount of the teacher model; The second training unit 440 is used to train the student model with minimizing the difference between the teacher's recognition result and the student's recognition result as the training goal, so as to obtain a semantic relationship recognition model.
[0085] Based on any of the above embodiments, the teacher training text is determined based on the following steps: Determine multiple candidate description texts related to target knowledge points, where the target knowledge points are knowledge points corresponding to the texts in the teacher training texts; Based on the semantic similarity between each candidate description text and the corresponding target knowledge point, a teacher training text is determined from multiple candidate description texts.
[0086] Based on any of the above embodiments, the multiple candidate description texts are determined based on the following steps: Based on the target knowledge point, a search is performed in the knowledge base to obtain a plurality of knowledge triples matching the target knowledge point; Based on a large language model, the semantic information of each knowledge triple is applied to perform retrieval enhancement on each knowledge triple, and a knowledge description text is generated to describe each knowledge triple; Based on the large language model, the knowledge description text of each knowledge triple is subjected to thought chain distillation prompts to obtain corresponding candidate description texts, and each candidate description text is used to describe the semantic relationship between two knowledge points in the corresponding knowledge triple in a thought chain manner.
[0087] Based on any of the above embodiments, determining a teacher training text from a plurality of candidate description texts based on the semantic similarity between each candidate description text and the corresponding target knowledge point includes: Based on the semantic similarity between each candidate description text and the corresponding target knowledge point, determining an initial training text from multiple candidate description texts; Based on the occurrence frequency and / or word frequency of the target keyword in each initial training text, the teacher training text is determined from each initial training text, and the target keyword is obtained by keyword extraction of each target knowledge point.
[0088] Based on any of the above embodiments, the student model is trained with minimizing the difference between the teacher's recognition result and the student's recognition result as the training goal to obtain a semantic relationship recognition model, including: The student model is trained with the training objectives of minimizing the difference between the teacher's recognition results and the student's recognition results, and minimizing the difference between the semantic relationship labels between the student's recognition results and each text in the student's training text, to obtain a semantic relationship recognition model.
[0089] Based on any of the above embodiments, a teacher model is trained based on the teacher training text, the semantic relationship labels between the texts in the teacher training text, and the recognition prompt text, including: Based on the initial model of the teacher model, the teacher training text and the recognition prompt text are applied to determine the teacher prediction results; Determine a training loss value based on the difference between the teacher's prediction results and the semantic relationship labels between the texts in the teacher's training texts; Based on the training loss value, the parameters of the initial model are fine-tuned to obtain the teacher model.
[0090] The semantic relationship recognition device provided by the present invention is described below. The semantic relationship recognition device described below and the semantic relationship recognition method described above can be referenced to each other.
[0091] Based on any of the above embodiments, Figure 5 is a schematic diagram of the structure of the semantic relationship recognition device provided by the present invention, such as Figure 5 As shown, the device comprises: A text determination unit 510, configured to determine a first target text and a second target text; A relationship recognition unit 520 is used to perform semantic relationship recognition on the first target text and the second target text based on a semantic relationship recognition model to obtain a semantic relationship recognition result; The semantic relationship recognition model is trained based on the semantic relationship recognition model training method described in any of the above embodiments.
[0092] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the semantic relationship recognition model training method, which includes: based on the teacher training text, the semantic relationship labels between the texts in the teacher training text and the recognition prompt text, training to obtain a teacher model, wherein the recognition prompt text is used to guide the teacher model to perform semantic relationship recognition on the texts in the teacher training text according to the thinking chain; based on the teacher model, performing semantic relationship recognition on the texts in the student training text to obtain a teacher recognition result; based on the student model, performing semantic relationship recognition on the texts in the student training text to obtain a student recognition result, wherein the parameter amount of the student model is less than the parameter amount of the teacher model; with the training goal of minimizing the difference between the teacher recognition result and the student recognition result, the student model is trained to obtain a semantic relationship recognition model.
[0093] Or execute a semantic relationship recognition method, the method comprising: determining a first target text and a second target text; based on a semantic relationship recognition model, performing semantic relationship recognition on the first target text and the second target text to obtain a semantic relationship recognition result.
[0094] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0095] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the semantic relationship recognition model training method provided by the above-mentioned methods, which method includes: based on a teacher training text, semantic relationship labels between each text in the teacher training text, and recognition prompt text, training a teacher model, wherein the recognition prompt text is used to guide the teacher model to perform semantic relationship recognition on each text in the teacher training text according to a thinking chain; based on the teacher model, performing semantic relationship recognition on each text in the student training text to obtain a teacher recognition result; based on the student model, performing semantic relationship recognition on each text in the student training text to obtain a student recognition result, wherein the parameter amount of the student model is less than the parameter amount of the teacher model; with the training goal of minimizing the difference between the teacher recognition result and the student recognition result, the student model is trained to obtain a semantic relationship recognition model.
[0096] Or execute a semantic relationship recognition method, the method comprising: determining a first target text and a second target text; based on a semantic relationship recognition model, performing semantic relationship recognition on the first target text and the second target text to obtain a semantic relationship recognition result.
[0097] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the semantic relationship recognition model training method provided by the above-mentioned methods, the method comprising: training a teacher model based on a teacher training text, semantic relationship labels between texts in the teacher training text, and recognition prompt text, wherein the recognition prompt text is used to guide the teacher model to perform semantic relationship recognition on each text in the teacher training text according to a thinking chain; based on the teacher model, performing semantic relationship recognition on each text in the student training text to obtain a teacher recognition result; based on the student model, performing semantic relationship recognition on each text in the student training text to obtain a student recognition result, wherein the number of parameters of the student model is less than the number of parameters of the teacher model; training the student model with the goal of minimizing the difference between the teacher recognition result and the student recognition result to obtain a semantic relationship recognition model.
[0098] Or execute a semantic relationship recognition method, the method comprising: determining a first target text and a second target text; based on a semantic relationship recognition model, performing semantic relationship recognition on the first target text and the second target text to obtain a semantic relationship recognition result.
[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A semantic relationship recognition model training method, characterized in that: include: Based on the teacher training text, the semantic relationship labels between the texts in the teacher training text, and the recognition prompt text, a teacher model is trained, wherein the recognition prompt text is used to guide the teacher model to perform semantic relationship recognition on the texts in the teacher training text according to the thinking chain; Based on the teacher model, semantic relationship recognition is performed on each text in the student training text to obtain a teacher recognition result; Based on the student model, semantic relationship recognition is performed on each text in the student training text to obtain a student recognition result, wherein the number of parameters of the student model is less than the number of parameters of the teacher model; The student model is trained with the training goal of minimizing the difference between the teacher's recognition result and the student's recognition result to obtain a semantic relationship recognition model.
2. The method for training a semantic relationship recognition model according to claim 1, characterized in that: The teacher training text is determined based on the following steps: Determine a plurality of candidate description texts related to a target knowledge point, wherein the target knowledge point is a knowledge point corresponding to each text in the teacher training text; Based on the semantic similarity between each candidate description text and the corresponding target knowledge point, the teacher training text is determined from the multiple candidate description texts.
3. The semantic relationship recognition model training method according to claim 2, characterized in that: The multiple candidate description texts are determined based on the following steps: Based on the target knowledge point, a search is performed in the knowledge base to obtain a plurality of knowledge triples matching the target knowledge point; Based on a large language model, the semantic information of each knowledge triple is applied to perform retrieval enhancement on each knowledge triple, and a knowledge description text is generated to describe each knowledge triple; Based on the large language model, the knowledge description text of each knowledge triple is subjected to thought chain distillation prompts to obtain corresponding candidate description texts, and each candidate description text is used to describe the semantic relationship between two knowledge points in the corresponding knowledge triple in a thought chain manner.
4. The semantic relationship recognition model training method according to claim 2, characterized in that: The step of determining the teacher training text from the plurality of candidate description texts based on the semantic similarity between each candidate description text and the corresponding target knowledge point comprises: Determining an initial training text from the plurality of candidate description texts based on the semantic similarity between each candidate description text and the corresponding target knowledge point; The teacher training text is determined from each initial training text based on the occurrence frequency and / or word frequency of the target keyword in each initial training text, and the target keyword is obtained by keyword extraction of each target knowledge point.
5. The method for training a semantic relationship recognition model according to any one of claims 1 to 4, characterized in that: The method of training the student model with minimizing the difference between the teacher recognition result and the student recognition result as the training goal to obtain a semantic relationship recognition model includes: The student model is trained with the training objectives of minimizing the difference between the teacher recognition result and the student recognition result, and minimizing the difference between the semantic relationship labels between the student recognition result and each text in the student training text, to obtain the semantic relationship recognition model.
6. The method for training a semantic relationship recognition model according to any one of claims 1 to 4, characterized in that: The training of a teacher model based on the teacher training text, the semantic relationship labels between the texts in the teacher training text, and the recognition prompt text includes: Based on the initial model of the teacher model, the teacher training text and the recognition prompt text are applied to determine the teacher prediction result; Determining a training loss value based on a difference between the teacher's prediction result and the semantic relationship labels between the texts in the teacher's training text; Based on the training loss value, the parameters of the initial model are fine-tuned to obtain the teacher model.
7. A semantic relationship recognition method, characterized in that: include: determining a first target text and a second target text; Based on the semantic relationship recognition model, performing semantic relationship recognition on the first target text and the second target text to obtain a semantic relationship recognition result; The semantic relationship recognition model is trained based on the semantic relationship recognition model training method described in any one of claims 1 to 6.
8. A semantic relationship recognition model training device, characterized in that: include: A first training unit is used to train a teacher model based on a teacher training text, semantic relationship labels between texts in the teacher training text, and recognition prompt text, wherein the recognition prompt text is used to guide the teacher model to perform semantic relationship recognition on each text in the teacher training text according to a thinking chain; A teacher identification unit, used to identify the semantic relationship of each text in the student training text based on the teacher model to obtain a teacher identification result; A student identification unit, used for performing semantic relationship identification on each text in the student training text based on a student model to obtain a student identification result, wherein the parameter amount of the student model is less than the parameter amount of the teacher model; The second training unit is used to train the student model with minimizing the difference between the teacher's recognition result and the student's recognition result as the training goal, so as to obtain a semantic relationship recognition model.
9. A semantic relationship recognition device, characterized in that: include: A text determination unit, used for determining a first target text and a second target text; A relationship recognition unit, configured to perform semantic relationship recognition on the first target text and the second target text based on a semantic relationship recognition model to obtain a semantic relationship recognition result; The semantic relationship recognition model is trained based on the semantic relationship recognition model training method described in any one of claims 1 to 6.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, it implements the semantic relationship recognition model training method as described in any one of claims 1 to 6, or the semantic relationship recognition method as described in claim 7.
11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the semantic relationship recognition model training method as described in any one of claims 1 to 6, or the semantic relationship recognition method as described in claim 7.