Accuracy discrimination method for generation content of large language model of engineering machinery
By combining the engineering machinery knowledge graph and knowledge base, we calculate the similarity of the content generated by large language models, and solve the problem of difficult to evaluate the accuracy of large language models generated by large language models in the field of engineering machinery, and achieve the accuracy and correctness of the content generated.
Patent Information
- Application Number
- CN202411395386.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2025-02-14
AI Technical Summary
The prior art is difficult to evaluate the accuracy of content generated by large language models in the field of engineering machinery, making it difficult for professionals to build confidence.
By obtaining the entities in the content generated by the construction machinery text input by the user, combining the construction machinery knowledge graph and knowledge base, the similarity between the generated content and the knowledge graph entity and knowledge base content is calculated, and the accuracy of the generated content is determined.
It effectively reduces the illusion problem of content generated by large language models, improves the accuracy of content generated, and ensures the correctness of generation results in the field of engineering machinery.
Smart Images

Figure CN119441901A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of engineering machinery, and in particular to a method for determining the accuracy of content generated by a large language model of engineering machinery, a system for determining the accuracy of content generated by a large language model of engineering machinery, an electronic device, and a machine-readable storage medium. Background Art
[0002] At present, the application of large language models in the field of engineering machinery is gradually expanding. However, due to the hallucination characteristics of large language models, that is, the content generated by the model does not exist in reality, it is difficult for professionals to build confidence in the results generated by large language models. Therefore, there is a lack of a method to judge the results generated by the model, and it is necessary to ensure the correctness of the results generated by the model. Summary of the invention
[0003] The purpose of the embodiments of the present application is to provide a method for determining the accuracy of content generated by a large language model of engineering machinery, a system for determining the accuracy of content generated by a large language model of engineering machinery, an electronic device, and a machine-readable storage medium, so as to partially or completely solve the problem in the prior art that it is difficult to evaluate the accuracy of the results generated by a large language model.
[0004] In order to achieve the above-mentioned purpose, the first aspect of the present application provides a method for determining the accuracy of content generated by a large language model of engineering machinery.
[0005] In an embodiment of the present application, the accuracy judgment method includes: obtaining a first entity in the generated content corresponding to the engineering machinery text input by the user; obtaining a second entity in the engineering machinery text input by the user; inputting the second entity into the engineering machinery knowledge graph to obtain a third entity with the highest similarity to the second entity and a fourth entity that is closest to the third entity; obtaining a first similarity between the third entity and the fourth entity and the first entity respectively; classifying the engineering machinery text input by the user according to a plurality of predefined categories, and matching the equipment knowledge corresponding to the classification result in the engineering machinery knowledge base according to the classification result; obtaining a second similarity between the equipment knowledge and the generated content; and determining the accuracy of the generated content based on the first similarity and the second similarity.
[0006] Optionally, the accuracy judgment method also includes: obtaining the category to which the first entity belongs based on the generated content corresponding to the engineering machinery text input by the user; obtaining the category to which the third entity belongs and the category to which the fourth entity belongs based on the third entity and the fourth entity; wherein, obtaining the first similarity between the third entity and the fourth entity and the first entity includes: obtaining the first similarity between the third entity and the fourth entity and the first entity based on the category to which the first entity belongs, the category to which the third entity belongs, and the category to which the fourth entity belongs.
[0007] Optionally, obtaining the first similarity between the third entity and the fourth entity and the first entity according to the category to which the first entity belongs, the category to which the third entity belongs, and the category to which the fourth entity belongs includes: calculating the similarity between the third entity and the fourth entity belonging to the same category and the first entity, respectively, to obtain the first similarity.
[0008] Optionally, determining the accuracy of the generated content according to the first similarity and the second similarity includes: determining the accuracy P of the generated content according to the following formula:
[0009]
[0010] Among them, P n Represents the first similarity, P x represents the second similarity, and m represents the total number of similarity calculations.
[0011] Optionally, the accuracy determination method includes: when the accuracy of the generated content is greater than or equal to a preset threshold, determining that the generated content is accurate.
[0012] Optionally, the accuracy determination method includes: when the accuracy of the generated content is less than a preset threshold, splicing the second entity with the equipment knowledge; using the spliced second entity and the equipment knowledge as new user-input engineering machinery text to obtain new generated content; and using the new generated content to re-obtain the accuracy of the new generated content until the accuracy of the new generated content is greater than or equal to the preset threshold.
[0013] According to a second aspect of the present application, there is provided an accuracy determination system for content generated by a large language model of engineering machinery, the accuracy determination system comprising: a first acquisition module, for acquiring a first entity in the generated content according to the generated content corresponding to the engineering machinery text input by the user; a second acquisition module, for acquiring a second entity in the engineering machinery text input by the user; a third acquisition module, for inputting the second entity into the engineering machinery knowledge graph, and acquiring a third entity with the highest similarity to the second entity and a fourth entity closest to the third entity; a fourth acquisition module, for acquiring a first similarity between the third entity and the fourth entity and the first entity respectively; a matching module, for classifying the engineering machinery text input by the user according to a plurality of predefined categories, and matching the equipment knowledge corresponding to the classification result in the engineering machinery knowledge base according to the classification result; a fifth acquisition module, for acquiring a second similarity between the equipment knowledge and the generated content; and a determination module, for determining the accuracy of the generated content according to the first similarity and the second similarity.
[0014] Furthermore, the accuracy determination system also includes: a sixth acquisition module, which acquires the category to which the first entity belongs according to the generated content corresponding to the engineering machinery text input by the user; a seventh module, which is used to acquire the category to which the third entity belongs and the category to which the fourth entity belongs according to the third entity and the fourth entity; wherein, the fourth acquisition module, which is used to acquire the first similarity between the third entity and the fourth entity and the first entity, includes: acquiring the first similarity between the third entity and the fourth entity and the first entity according to the category to which the first entity belongs, the category to which the third entity belongs, and the category to which the fourth entity belongs.
[0015] The third aspect of the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method for determining the accuracy of content generated by the large language model of engineering machinery.
[0016] A fourth aspect of the present application provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute a method for determining the accuracy of content generated according to the large language model of engineering machinery.
[0017] Through the above technical scheme, the present invention inputs the engineering machinery text input by the user into the engineering machinery large language model to obtain generated content (such as question and answer results), and further obtains the first entity in the generated content; at the same time, the second entity in the engineering machinery text input by the user is obtained, and the entity is input into the engineering machinery knowledge graph to obtain the third entity with the highest similarity to the second entity and the fourth entity closest to the third entity, and further obtain the first similarity between the third entity and the fourth entity and the first entity respectively; the engineering machinery text input by the user is classified according to multiple predefined categories to obtain the corresponding equipment knowledge, and further obtain the second similarity between the equipment knowledge and the generated content; finally, the accuracy of the generated content is evaluated according to the first similarity and the second similarity.
[0018] Therefore, the embodiment of the present invention calculates the similarity with the engineering machinery knowledge graph entities, combines the calculation of similarity between the generated content and the knowledge base content, and then uses the comprehensive consideration of these two similarity calculations to evaluate the accuracy of the generated content, thereby further reducing the hallucination problem of the generated content of the engineering machinery large model and ensuring the accuracy of the generated content of the engineering large language model.
[0019] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:
[0021] Figure 1 It is a flowchart of a method for determining the accuracy of generated content according to an embodiment of the present invention;
[0022] Figure 2 It is an overall flow chart of the method for determining the accuracy of generated content provided by the implementation of the present invention;
[0023] Figure 3 It is a structural schematic diagram of an accuracy determination system provided by the implementation of the present invention. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0025] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0026] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0027] Figure 1 The flowchart schematically shows a method for determining the accuracy of content generated by a large language model of an engineering machinery according to an embodiment of the present application. Figure 1 As shown, the embodiment of the present application provides a method for determining the accuracy of content generated by a large language model of engineering machinery, and the method may include the following steps 101-107. Figure 2 , Figure 2 It is an overall flow chart of the accuracy determination method of the content generated by the large language model of engineering machinery provided by the implementation of the present invention.
[0028] Step 101: According to generated content corresponding to the engineering machinery text input by the user, obtain a first entity in the generated content.
[0029] In the embodiment of the present application, the embodiment of the present invention is based on the premise that the user inputs the engineering text into the engineering machinery training model and then obtains the generated content, wherein the so-called generated content is the question and answer result generated by the large model. The corresponding generated content can be obtained by simply inputting the user input engineering machinery text into the trained engineering machinery training model.
[0030] The text input by the user for engineering machinery can be equipment data query, equipment fault query or equipment maintenance query. The above input content will pass through the engineering machinery large language model and obtain the generated result corresponding to the input. For example, for equipment fault query, the user inputs the engineering machinery text "the boom cannot be lifted up", and the corresponding generated content (or question and answer result) is "the boom does not move during use". The subsequent steps of the embodiment of the present invention are mainly used to evaluate whether the generated content is accurate.
[0031] Step 102: Obtain a second entity in the engineering machinery text input by the user.
[0032] In the embodiment of the present application, the generated content of the large model of engineering machinery is identified using a trained engineering machinery entity recognition model, and all engineering machinery-related entities contained in the generated content are identified. The entity obtained is the second entity, and the content that is not related to the entity in this process will be discarded after entity recognition. For example: "The boom has no action during use", and the result obtained after entity recognition is "{"part": "boom"}, {"error": "no action"}", where part refers to the faulty component and error refers to the fault type.
[0033] Step 103: input the second entity into the engineering machinery knowledge graph to obtain the third entity with the highest similarity to the second entity and the fourth entity that is closest to the third entity.
[0034] In an embodiment of the present application, the second entity obtained in step 102 is used as input in the engineering machinery knowledge graph. There are a large number of trained entities related to engineering machinery in the engineering machinery knowledge graph. Then the second entity is matched with the entities in the knowledge graph. After matching, the third entity with the highest similarity (highest matching probability) in the knowledge graph and the fourth entity closest to the third entity will be obtained (the knowledge graph is stored in the form of nodes and edges, and the closest fourth entity refers to the entity directly connected to the third entity through an edge).
[0035] Step 104: Obtain a first similarity between the third entity and the fourth entity and the first entity.
[0036] In the embodiment of the present application, this step obtains the similarity between the third entity and the first entity, and the similarity between the fourth entity and the first entity through existing technical means, which is used to provide an important theoretical basis for the subsequent comprehensive judgment of the similarity in the generated content.
[0037] Furthermore, the accuracy determination method also includes steps 201-202.
[0038] Step 201: Obtain the category to which the first entity belongs according to the generated content corresponding to the engineering machinery text input by the user.
[0039] Step 202: According to the third entity and the fourth entity, obtain the category to which the third entity belongs and the category to which the fourth entity belongs.
[0040] In the embodiment of the present application, the above steps 201 and 202 both use the same method to obtain the category to which the entity belongs, and the category may specifically include but is not limited to fault type, component, maintenance information, etc.
[0041] The obtaining of the first similarity between the third entity and the fourth entity and the first entity includes: obtaining the first similarity between the third entity and the fourth entity and the first entity according to the category to which the first entity belongs, the category to which the third entity belongs, and the category to which the fourth entity belongs.
[0042] Further, obtaining the first similarity between the third entity and the fourth entity and the first entity according to the category to which the first entity belongs, the category to which the third entity belongs, and the category to which the fourth entity belongs includes: calculating the similarity between the third entity and the fourth entity belonging to the same category and the first entity, respectively, to obtain the first similarity.
[0043] In an embodiment of the present application, the third entity and the fourth entity belonging to the same category are respectively calculated for similarity with the first entity. For example: the first entity belongs to 2 entities of the part category and 4 entities of the error category. The third entity belongs to 3 entities of the part category and 2 entities of the error category. The fourth entity belongs to 2 entities of the part category and 2 entities of the error category. Then all entities of the part category will undergo 2×3+2×2=10 similarity calculations, and errors will undergo 4×2+4×2=16 similarity calculations, so a total of 26 similarity calculations are performed, corresponding to a total of 26 first similarities.
[0044] Step 105 : classify the engineering machinery text input by the user according to a plurality of predefined categories, and match the equipment knowledge corresponding to the classification result in the engineering machinery knowledge base according to the classification result.
[0045] In the embodiment of the present application, multiple predefined categories are provided, such as major fault categories and components, where the major fault categories can be further divided into boom inaction, outrigger immobility, etc., and components include but are not limited to booms and outriggers. The predefined categories can be set according to actual needs, and there are no specific restrictions on the categories and quantities. Then, the user input engineering machinery text is classified according to the multiple predefined categories, and corresponding classification results are obtained.
[0046] Equipment knowledge corresponds to the classification results, such as the fault type, maintenance type, working condition, etc. For example, if the user enters the text of "the boom cannot be lifted" for construction machinery, it will be classified as "the boom does not move" under the fault type, where "the boom does not move" also corresponds to the specific "fault type" under "equipment knowledge".
[0047] Step 106: Obtain a second similarity between the device knowledge and the generated content.
[0048] In the embodiment of the present application, the similarity between the device knowledge and the generated content is also obtained according to the existing technical means, and the embodiment of the present invention does not specifically limit the similarity calculation method.
[0049] Step 107: Determine the accuracy of the generated content according to the first similarity and the second similarity.
[0050] Determining the accuracy of the generated content according to the first similarity and the second similarity includes: determining the accuracy P of the generated content according to the following formula:
[0051] Among them, P n Represents the first similarity, P x represents the second similarity, and m represents the total number of similarity calculations.
[0052] In the embodiment of the present application, the first similarity is first averaged, and then weighted summed with the second similarity to obtain the accuracy P calculation value, which can be used to evaluate whether the generated content (or question and answer result) is accurate. Where m is the total value 26 calculated in the above steps.
[0053] The embodiment of the present invention mainly uses TongYi-7B in natural language processing as the generated base model, uses the cosine similarity method to judge the entity similarity, uses the Bert twin network to judge the similarity between the generated content and the knowledge base content, and uses Bert to perform text classification.
[0054] However, the actual natural language processing algorithm can also use chatgpt, chatglm and other generation models, and use other classification algorithms such as Text-CNN and Text-RNN to achieve similar functions. After building the same structure according to the present invention, the final result similar to the present invention can be achieved.
[0055] Compared with the existing retrieval enhancement generation method to reduce the hallucination of large language models, this patent can obtain more accurate results by combining the knowledge graph entity similarity calculation with the similarity judgment between the generated content and the knowledge base content and weighted calculation, thereby further reducing the hallucination problem of the generated content of large models of engineering machinery.
[0056] Further, the accuracy determination method includes: when the accuracy of the generated content is greater than or equal to a preset threshold, determining that the generated content is accurate.
[0057] In the embodiment of the present application, if the preset threshold is 0.8, then when the accuracy of the generated content is greater than or equal to 0.8, it means that the generation result of the large language model of engineering machinery is accurate.
[0058] Furthermore, the accuracy determination method also includes steps 301-303.
[0059] Step 301: When the accuracy of the generated content is less than a preset threshold, the second entity is spliced with the device knowledge.
[0060] Step 302: Use the spliced second entity and the equipment knowledge as new user-input engineering machinery text to obtain new generated content.
[0061] Step 303: Re-acquire the accuracy of the newly generated content using the newly generated content until the accuracy of the newly generated content is greater than or equal to a preset threshold.
[0062] In the embodiment of the present application, when the accuracy of the generated content is less than 0.8, it means that the output result of the large model is inaccurate. Therefore, in order to further obtain the generated content that meets the requirements, the second entity is spliced with the equipment knowledge as a new user input engineering machinery text to obtain new generated content. Then execute the above steps 101-107 and judge whether the new generated content meets the accuracy requirements. The process is executed repeatedly until the generated content meets or equals 0.8, thereby obtaining the generated content that meets the requirements (i.e., the question and answer results).
[0063] The input content is completed by combining similar entities in the graph with the knowledge base content, ensuring the accuracy of the input results. The accuracy of the generated results of the large language model of engineering machinery is tested by judging the similarity between the content generated by the large model and the content of the knowledge base, combining the similarity judgment between the content generated by the large model and the graph content, ensuring the correctness of the generated results in the field of engineering machinery.
[0064] For erroneous content generated when hallucinations exist, similar entities and nearest neighbors in the knowledge graph are used to supplement it, and the knowledge base content obtained by classification calculations is used to supplement it. This increases the comprehensiveness of the input and further prevents the model from hallucinating when generating content in professional fields.
[0065] This patent invents a method for determining the accuracy of content generated by a large language model for engineering machinery. The method uses a knowledge graph to verify the generated results of the input text after passing through the large language model using the knowledge graph similarity entity and knowledge base content similarity judgment method. At the same time, for dissimilar content, the knowledge graph entities and knowledge base content are used to supplement the user input to ensure the accuracy of the generated results.
[0066] The embodiment of the present invention calculates the similarity with the engineering machinery knowledge graph entities, combines the calculation of similarity between the generated content and the knowledge base content, and then uses the comprehensive consideration of these two similarity calculations to evaluate the accuracy of the generated content, thereby further reducing the hallucination problem of the generated content of the engineering machinery large model and ensuring the accuracy of the generated content of the engineering large language model.
[0067] Figure 3 The following schematically shows a structural diagram of a system for determining the accuracy of content generated by a large language model of an engineering machinery according to an embodiment of the present application. Figure 3 As shown, in the embodiment of the present application, the accuracy determination system 30 includes: a first acquisition module 301, which is used to acquire a first entity in the generated content according to the generated content corresponding to the user-input engineering machinery text; a second acquisition module 302, which is used to acquire a second entity in the user-input engineering machinery text; a third acquisition module 303, which is used to input the second entity into the engineering machinery knowledge graph, and acquire a third entity with the highest similarity to the second entity and a fourth entity that is closest to the third entity; a fourth acquisition module 304, which is used to acquire a first similarity between the third entity and the fourth entity and the first entity respectively; a matching module 305, which is used to classify the user-input engineering machinery text according to a plurality of predefined categories, and match the equipment knowledge corresponding to the classification result in the engineering machinery knowledge base according to the classification result; a fifth acquisition module 306, which is used to acquire a second similarity between the equipment knowledge and the generated content; a determination module 307, which is used to determine the accuracy of the generated content according to the first similarity and the second similarity.
[0068] The accuracy determination system also includes: a sixth acquisition module, which acquires the category to which the first entity belongs according to the generated content corresponding to the engineering machinery text input by the user; a seventh module, which is used to acquire the category to which the third entity belongs and the category to which the fourth entity belongs according to the third entity and the fourth entity; wherein the fourth acquisition module, which is used to acquire the first similarity between the third entity and the fourth entity and the first entity, includes: acquiring the first similarity between the third entity and the fourth entity and the first entity according to the category to which the first entity belongs, the category to which the third entity belongs, and the category to which the fourth entity belongs.
[0069] An embodiment of the present invention also includes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the accuracy determination method for content generated by a large language model of engineering machinery as described above.
[0070] The embodiment of the present invention also includes a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the accuracy determination method for content generated by the large language model of engineering machinery.
[0071] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0072] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0073] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0075] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0076] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0077] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0078] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0079] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
Claims
1. A method for determining the accuracy of content generated by a large language model of engineering machinery, characterized in that: The accuracy determination method comprises: According to the generated content corresponding to the engineering machinery text input by the user, a first entity in the generated content is obtained; Acquire a second entity in the engineering machinery text input by the user; Input the second entity into the engineering machinery knowledge graph to obtain a third entity with the highest similarity to the second entity and a fourth entity that is closest to the third entity; Obtaining a first similarity between the third entity and the fourth entity and the first entity; Classifying the engineering machinery text input by the user according to a plurality of predefined categories, and matching the equipment knowledge corresponding to the classification result in the engineering machinery knowledge base according to the classification result; Acquire a second similarity between the device knowledge and the generated content; The accuracy of the generated content is determined according to the first similarity and the second similarity.
2. The accuracy determination method according to claim 1, characterized in that: The accuracy determination method further includes: obtaining the category to which the first entity belongs according to the generated content corresponding to the engineering machinery text input by the user; According to the third entity and the fourth entity, acquiring a category to which the third entity belongs and a category to which the fourth entity belongs; The obtaining of a first similarity between the third entity and the fourth entity and the first entity includes: According to the category to which the first entity belongs, the category to which the third entity belongs, and the category to which the fourth entity belongs, a first similarity between the third entity and the fourth entity and the first entity is obtained.
3. The accuracy determination method according to claim 2, characterized in that: The acquiring, according to the category to which the first entity belongs, the category to which the third entity belongs, and the category to which the fourth entity belongs, a first similarity between each of the third entity and the first entity comprises: A third entity and a fourth entity belonging to the same category are respectively subjected to similarity calculations with the first entity to obtain a first similarity.
4. The accuracy determination method according to claim 1, characterized in that: Determining the accuracy of the generated content according to the first similarity and the second similarity includes: The accuracy P of the generated content is determined according to the following formula: Among them, P n Represents the first similarity, P x represents the second similarity, and m represents the total number of similarity calculations.
5. The accuracy determination method according to claim 1, characterized in that: The accuracy determination method comprises: When the accuracy of the generated content is greater than or equal to a preset threshold, it is determined that the generated content is accurate.
6. The accuracy determination method according to claim 5, characterized in that: The accuracy determination method further comprises: When the accuracy of the generated content is less than a preset threshold, splicing the second entity with the device knowledge; The spliced second entity and the equipment knowledge are used as new user input engineering machinery text to obtain new generated content; The accuracy of the newly generated content is reacquired using the newly generated content until the accuracy of the newly generated content is greater than or equal to a preset threshold.
7. A system for determining the accuracy of content generated by a large language model of construction machinery, characterized in that: The accuracy determination system comprises: A first acquisition module, configured to acquire a first entity in the generated content according to the generated content corresponding to the engineering machinery text input by the user; A second acquisition module, used to acquire a second entity in the engineering machinery text input by the user; A third acquisition module, used to input the second entity into the engineering machinery knowledge graph, and acquire a third entity with the highest similarity to the second entity and a fourth entity that is most adjacent to the third entity; A fourth acquisition module, configured to acquire a first similarity between the third entity and the fourth entity and the first entity; A matching module, used to classify the engineering machinery text input by the user according to a plurality of predefined categories, and match the equipment knowledge corresponding to the classification result in the engineering machinery knowledge base according to the classification result; A fifth acquisition module, configured to acquire a second similarity between the device knowledge and the generated content; A determination module is used to determine the accuracy of the generated content according to the first similarity and the second similarity.
8. The accuracy determination system according to claim 7, characterized in that: The accuracy determination system also includes: A sixth acquisition module, acquiring the category to which the first entity belongs according to the generated content corresponding to the engineering machinery text input by the user; A seventh module is used to obtain, according to the third entity and the fourth entity, a category to which the third entity belongs and a category to which the fourth entity belongs; The fourth acquisition module, for acquiring the first similarity between the third entity and the fourth entity and the first entity, comprises: According to the category to which the first entity belongs, the category to which the third entity belongs, and the category to which the fourth entity belongs, a first similarity between the third entity and the fourth entity and the first entity is obtained.
9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the accuracy determination method for content generated by a large language model of engineering machinery according to any one of claims 1 to 6.
10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for enabling a machine to execute the method for determining the accuracy of content generated by a large language model of engineering machinery according to any one of claims 1 to 6.
