Method and device for evaluating large semantic model in electric power professional scene

Abstract

The invention provides an evaluation method and device for a semantic large model in an electric power professional scene, and belongs to the field of electric power models. The method comprises the following steps: constructing an evaluation question bank of a to-be-evaluated electric power major semantic large model; inputting the evaluation question bank into an electric power major semantic large model to obtain a first semantic vector; determining a second semantic vector corresponding to the evaluation question bank based on the evaluation question bank and a pre-constructed knowledge graph of the electric power professional field; and based on the correlation degree of the first semantic vector and the second semantic vector, evaluating the semantic large model in the electric power professional scene. According to the method, the accuracy and objectivity of large semantic model evaluation in an electric power professional scene can be improved.

Description

Technical Field

[0001] The present invention relates to the field of power model technology, and in particular to a method and device for evaluating a semantic large model in a power professional scenario. Background Art

[0002] In recent years, with the increasing severity of climate change and sustainable development issues, reducing greenhouse gas emissions has become a top priority. The new power system is based on renewable energy, which helps to achieve energy conservation and emission reduction. However, the new power system faces many challenges, such as power supply security, new energy consumption, and system operation pressure.

[0003] With the rapid development of artificial intelligence (AI) technology, especially the application of semantic big models in the power field is becoming more and more extensive. The semantic big model is a parameterized knowledge base that builds a powerful speech model by learning massive data and has strong semantic understanding capabilities. Learning ability and creative ability have strong versatility at the application level. In the power industry, big models have begun to be integrated into the links from power production to consumption. Typical application scenarios include equipment quality inspection, transmission line inspection, and marketing. For the power industry, semantic big models will become a new basic setting.

[0004] However, due to the lack of professional knowledge and insufficient timeliness of knowledge update, the output results of large models often have problems of instability or lack of professionalism. At present, the performance evaluation standards for large semantic models in power professional scenarios are not very objective, and most evaluation methods are manual evaluation, that is, the results of the large language model are fed back to the R&D personnel, who manually judge whether the results are reasonable. Manually designing evaluation indicators is time-consuming and laborious, and may have certain subjectivity and limitations. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for evaluating a semantic big model in a power professional scenario to solve the problem that the current evaluation methods have certain subjectivity and limitations.

[0006] In a first aspect, an embodiment of the present invention provides an evaluation method for a semantic big model in a power professional scenario, including:

[0007] Construct an evaluation question bank of the semantic large model of the power professional to be evaluated;

[0008] Input the evaluation question bank into the electric power professional semantic big model to obtain the first semantic vector;

[0009] Based on the evaluation question bank and the pre-constructed knowledge graph of the electric power professional field, determine a second semantic vector corresponding to the evaluation question bank;

[0010] Based on the correlation between the first semantic vector and the second semantic vector, the semantic big model in the power professional scenario is evaluated.

[0011] In a possible implementation, based on the evaluation question bank and the pre-built knowledge graph in the electric power professional field, determining the second semantic vector corresponding to the evaluation question bank includes:

[0012] Determine the edge relationships and associated nodes of the evaluation question bank in the pre-constructed knowledge graph of the power professional field;

[0013] Based on the edge relationship and the associated nodes, a second semantic vector corresponding to the evaluation question bank is determined.

[0014] In a possible implementation, based on the correlation between the first semantic vector and the second semantic vector, the semantic big model in the power professional scenario is evaluated, including:

[0015] Determining a first similarity based on word similarities between the first semantic vector and the second semantic vector;

[0016] Determining a second similarity based on the short text similarity between the first semantic vector and the second semantic vector;

[0017] Determining a third similarity based on the ROUGE values ​​of the first semantic vector and the second semantic vector;

[0018] Based on the first similarity, the second similarity and the third similarity, the semantic big model in the power professional scenario is evaluated.

[0019] In a possible implementation, the semantic big model in the electric power professional scenario is evaluated based on the first similarity, the second similarity and the third similarity, including:

[0020] Determining a similarity index based on a sum of the first similarity, the second similarity, and the third similarity;

[0021] Based on the similarity index and the preset similarity index evaluation standard, the semantic big model in the power professional scenario is evaluated.

[0022] In a possible implementation, the semantic big model in the electric power professional scenario is evaluated based on the first similarity, the second similarity and the third similarity, including:

[0023] Determining a similarity index based on a product of the first similarity, the second similarity, and the third similarity;

[0024] Based on the similarity index and the preset similarity index evaluation standard, the semantic big model in the power professional scenario is evaluated.

[0025] In a possible implementation, the semantic big model in the power professional scenario is evaluated based on the similarity index and the preset similarity index evaluation standard, including:

[0026] Determine a similarity difference based on the similarity index and a preset evaluation threshold;

[0027] Determine the level interval corresponding to the similarity difference value based on the preset similarity index interval in which the similarity difference value is located;

[0028] Based on the scores corresponding to the level intervals, the semantic big model in the power professional scenario is evaluated.

[0029] In a possible implementation, based on the evaluation question bank and the pre-built knowledge graph in the electric power professional field, determining the second semantic vector corresponding to the evaluation question bank includes:

[0030] Build a knowledge graph in the power industry through a crowdsourcing platform;

[0031] Build an entity synonym forest in the power industry based on the bootstrapping method;

[0032] Link to the knowledge graph in the power professional field based on the entity synonym forest;

[0033] A matching query is performed on the evaluation question bank in the knowledge graph of the electric power professional field to determine the second semantic vector.

[0034] In one possible implementation, the evaluation question bank of the electric power professional semantic big model is obtained based on a survey of various application scenarios in the electric power industry and tested with multiple models.

[0035] In a second aspect, an embodiment of the present invention provides an evaluation device for a semantic big model in a power professional scenario, including:

[0036] A construction module is used to construct an evaluation question bank of the electric power professional semantic large model to be evaluated;

[0037] A first determination module is used to input the evaluation question bank into the electric power professional semantic big model to obtain a first semantic vector;

[0038] A second determination module is used to determine a second semantic vector corresponding to the evaluation question bank based on the evaluation question bank and a pre-constructed knowledge graph in the electric power professional field;

[0039] The evaluation module is used to evaluate the semantic big model in the power professional scenario based on the correlation between the first semantic vector and the second semantic vector.

[0040] In a possible implementation, the second determination module is used to determine the edge relationships and associated nodes of the evaluation question bank in the pre-constructed knowledge graph of the power professional field;

[0041] Based on the edge relationship and the associated nodes, a second semantic vector corresponding to the evaluation question bank is determined.

[0042] In a possible implementation, the evaluation module is used to determine the first similarity based on the word similarity between the first semantic vector and the second semantic vector;

[0043] Determining a second similarity based on the short text similarity between the first semantic vector and the second semantic vector;

[0044] Determining a third similarity based on the ROUGE values ​​of the first semantic vector and the second semantic vector;

[0045] Based on the first similarity, the second similarity and the third similarity, the semantic big model in the power professional scenario is evaluated.

[0046] In a possible implementation, the evaluation module is used to determine a similarity index based on the sum of the first similarity, the second similarity, and the third similarity;

[0047] Based on the similarity index and the preset similarity index evaluation standard, the semantic big model in the power professional scenario is evaluated.

[0048] In a possible implementation, the evaluation module is used to determine a similarity index based on a product of the first similarity, the second similarity, and the third similarity;

[0049] Based on the similarity index and the preset similarity index evaluation standard, the semantic big model in the power professional scenario is evaluated.

[0050] In a possible implementation, the evaluation module is used to determine the similarity difference based on the similarity index and a preset evaluation threshold;

[0051] Determine the level interval corresponding to the similarity difference value based on the preset similarity index interval in which the similarity difference value is located;

[0052] Based on the scores corresponding to the level intervals, the semantic big model in the power professional scenario is evaluated.

[0053] In a possible implementation, the second determination module is used to construct a knowledge graph in the electric power professional field through a crowdsourcing platform;

[0054] Build an entity synonym forest in the power industry based on the bootstrapping method;

[0055] Link to the knowledge graph in the power professional field based on the entity synonym forest;

[0056] A matching query is performed on the evaluation question bank in the knowledge graph of the electric power professional field to determine the second semantic vector.

[0057] In one possible implementation, the evaluation question bank of the electric power professional semantic big model is obtained based on a survey of various application scenarios in the electric power industry and tested with multiple models.

[0058] The embodiment of the present invention provides an evaluation method and device for a semantic big model in an electric power professional scenario. In order to evaluate the electric power professional semantic big model to be evaluated more objectively, an evaluation question bank of the electric power professional semantic big model to be evaluated is first constructed. Then, the evaluation question bank is input into the electric power professional semantic big model to obtain a first semantic vector. The first semantic vector can reflect the performance of the electric power professional semantic big model to be evaluated. Then, based on the evaluation question bank and the pre-constructed knowledge graph of the electric power professional field, a second semantic vector corresponding to the evaluation question bank is determined. Based on the evaluation question bank and the pre-constructed knowledge graph of the electric power professional field, a second semantic vector corresponding to the knowledge graph can be output through the knowledge graph. Finally, by determining the correlation between the first semantic vector and the second semantic vector, the evaluation of the semantic big model in the electric power professional scenario is completed. The evaluation method provided by the present invention can not only objectively and accurately evaluate the semantic big model in the electric power professional scenario, but also improve the speed of evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 It is a flowchart of an implementation of a method for evaluating a semantic big model in a power professional scenario provided by an embodiment of the present invention;

[0061] Figure 2 It is a block diagram of an evaluation method of a semantic big model in a power professional scenario provided by an embodiment of the present invention;

[0062] Figure 3 It is a structural diagram of an evaluation device for a semantic big model in a power professional scenario provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0064] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0065] As introduced in the background technology, the application prospects of large models in the power industry are very broad, and they can significantly improve the operating efficiency, safety and intelligence level of the power system. Compared with the traditional small power model, the power semantic large model has the following advantages:

[0066] First, it has stronger learning ability and can adapt to more complex scenarios. Second, it has stronger generalization ability and can be designed in one place and reused in many places. Third, it has stronger creative ability and emerges with stronger intelligent performance. Fourth, it has better performance and effect, and the model is more accurate.

[0067] Figure 1 and Figure 2 The implementation flow chart of the evaluation method of the semantic big model in the electric power professional scenario provided by the embodiment of the present invention is described in detail as follows:

[0068] S110. Construct an evaluation question bank of the semantic big model of the electric power profession to be evaluated.

[0069] Since the electric power professional semantic big model involves multiple aspects, in order to be able to evaluate the electric power professional semantic big model more objectively, it is necessary to construct a corresponding evaluation question bank.

[0070] In some embodiments, the evaluation question bank covers professional knowledge in the power field, common sense encyclopedic knowledge in the power field, power field safety and other knowledge, and can be used to evaluate the power professional semantic big model.

[0071] In this embodiment, a preliminary evaluation question bank of the electric power professional semantic big model is constructed by investigating various application scenarios in the electric power industry; multi-model testing and manual evaluation are performed on the evaluation question bank of the preliminary electric power professional semantic big model; based on the multi-model testing results and manual evaluation results, the preliminary electric power professional semantic big model evaluation question bank is modified to obtain a modified electric power field language model evaluation question bank; multi-model testing and manual evaluation are performed on the modified electric power professional semantic big model evaluation question bank until the answer accuracy of the big language model after fine-tuning in the electric power field reaches the set value, and there is no modification suggestion in the manual evaluation, and the modified electric power professional semantic big model evaluation question bank is used as the electric power professional semantic big model evaluation question bank.

[0072] S120, input the evaluation question bank into the electric power professional semantic big model to obtain the first semantic vector.

[0073] The evaluation question bank can be input into the electric power professional semantic big model, which can process each sentence in the evaluation question bank, obtain the vector of each sentence, and fuse the vectors of each sentence to obtain the first semantic vector corresponding to the evaluation question bank.

[0074] In some embodiments, the electric power professional semantic big model can perform word segmentation processing on the evaluation question bank, obtain each word segmentation contained in the evaluation question bank, and determine the vector of each word segmentation based on each word segmentation, and finally merge all the vectors to obtain the first semantic vector corresponding to the evaluation question bank.

[0075] S130. Based on the evaluation question bank and a pre-constructed knowledge graph in the electric power professional field, determine a second semantic vector corresponding to the evaluation question bank.

[0076] The knowledge graph is essentially a semantic network, a graph-based data structure consisting of nodes (Point) and edges (Edge). In the knowledge graph, each node represents an "entity" that exists in the real world, and each edge is a "relationship" between entities. The knowledge graph is the most effective way to represent relationships.

[0077] In some embodiments, it is necessary to first construct a knowledge graph in the power professional field, and the construction process is as follows:

[0078] S1310. Build a knowledge graph in the power industry through a crowdsourcing platform.

[0079] Crowdsourcing platforms can use appropriate human intervention in the construction of knowledge graphs, thereby effectively alleviating the problem that it is difficult to achieve high accuracy and high coverage of knowledge graphs by relying solely on data-driven methods. Crowdsourcing platforms connect a group of loose task issuers and task completers to achieve a series of operations such as task contracting matching.

[0080] S1320. Construct an entity synonym forest in the electric power professional field based on the bootstrap method.

[0081] First, obtain the corpus and define the initial schema.

[0082] Next, define the pattern for synonym extraction.

[0083] Thirdly, pattern discovery is performed, that is, each new synonym is searched in the corpus to find new patterns.

[0084] Then, the new patterns found are added to the existing pattern library and marked as "new patterns". When all new patterns have been added to the pattern library, all patterns are formed.

[0085] The new pattern is added to the existing pattern library and pattern matching is performed again to obtain new synonym pairs, wherein the new pattern refers to all new patterns, and the above matching steps are repeated until the termination condition is reached, that is, the accuracy reaches a certain level, or no more synonym pairs can be found.

[0086] S1330. Link to the knowledge graph in the electric power professional field based on the entity synonym forest.

[0087] Entity linking, also known as entity disambiguation, is the key to entity understanding and is the process of connecting entity references in text to specific entities in the knowledge graph.

[0088] S1340. Perform a matching query on the evaluation question bank in the knowledge graph in the electric power professional field to determine the second semantic vector.

[0089] By matching the evaluation question bank with the knowledge graph in the electric power professional field, a second semantic vector corresponding to the evaluation question bank can be output.

[0090] In some embodiments, first, the edge relationships and associated nodes of the evaluation question bank in the pre-constructed knowledge graph of the electric power professional field can be determined.

[0091] Then, based on the edge relationship and the associated nodes, a second semantic vector corresponding to the evaluation question bank is determined.

[0092] S140. Evaluate the semantic big model in the electric power professional scenario based on the correlation between the first semantic vector and the second semantic vector.

[0093] In some embodiments, the obtained first semantic vector and second semantic vector may be evaluated by word similarity, short text similarity and ROUGE value.

[0094] Word similarity is an important concept in natural language processing, which involves measuring the semantic proximity of two words. This similarity can be calculated based on a variety of methods and techniques, including but not limited to methods based on semantic distance and methods based on statistical models. The semantic distance-based method mainly calculates the similarity between two words by comparing the semantic differences between them. Among them, commonly used semantic distances include path length, information entropy, cosine similarity, etc. The statistical model-based method calculates the similarity between them by analyzing the context of words in a large model corpus.

[0095] Short text similarity is to find the similarity between two short texts. It is a special form of text matching tasks or text implication tasks, and returns the specific numerical value of the similarity between texts. It refers to the similarity calculation of short texts (generally less than 50 characters in Chinese). The larger the output value, the higher the semantic similarity. For example, in the question-answering system task (question-answering robot), we often manually configure some commonly used and clearly described questions and their corresponding answers. We call these configured questions "standard questions". When a user asks a question, the user's question is often compared with all the configured standard questions for similarity calculation, and the standard question that is most similar to the user's question is found and its answer is returned to the user, thus completing a question-answering operation. Short text similarity algorithms can be divided into three categories: (1) unsupervised similarity calculation; (2) supervised similarity calculation; (3) supervised + unsupervised similarity calculation.

[0096] The ROUGE value is proposed to solve the matching problem between the generated text and the reference text in generation tasks such as machine translation. It measures the number of overlapping words or characters between the candidate text and the reference text.

[0097] The specific evaluation process is as follows:

[0098] S1410: Determine a first similarity based on word similarities between the first semantic vector and the second semantic vector.

[0099] S1420: Determine a second similarity based on the short text similarity between the first semantic vector and the second semantic vector.

[0100] S1430 : Determine a third similarity based on the ROUGE values ​​of the first semantic vector and the second semantic vector.

[0101] S1440. Evaluate the semantic big model in the electric power professional scenario based on the first similarity, the second similarity and the third similarity.

[0102] In some embodiments, a similarity index may be first determined based on the sum of the first similarity, the second similarity, and the third similarity.

[0103] Then, based on the similarity index and the preset similarity index evaluation criteria, the semantic big model in the power professional scenario is evaluated.

[0104] In this embodiment, the similarity difference value may be determined first based on the similarity index and a preset evaluation threshold.

[0105] Then, based on the preset similarity index interval in which the similarity difference value is located, the level interval corresponding to the similarity difference value is determined.

[0106] Finally, based on the scores corresponding to the level intervals, the semantic big model in the power professional scenario is evaluated.

[0107] In some embodiments, a similarity index may be first determined based on the product of the first similarity, the second similarity, and the third similarity.

[0108] Then, based on the similarity index and the preset similarity index evaluation criteria, the semantic big model in the power professional scenario is evaluated.

[0109] In this embodiment, the similarity difference value may be determined first based on the similarity index and a preset evaluation threshold.

[0110] Then, based on the preset similarity index interval in which the similarity difference value is located, the level interval corresponding to the similarity difference value is determined.

[0111] Finally, based on the scores corresponding to the level intervals, the semantic big model in the power professional scenario is evaluated.

[0112] The evaluation method provided by the present invention, in order to be able to more objectively evaluate the electric power professional semantic big model to be evaluated, first constructs an evaluation question bank of the electric power professional semantic big model to be evaluated. Then, the evaluation question bank is input into the electric power professional semantic big model to obtain a first semantic vector. The first semantic vector can reflect the performance of the electric power professional semantic big model to be evaluated. Then, based on the evaluation question bank and the pre-constructed knowledge graph of the electric power professional field, the second semantic vector corresponding to the evaluation question bank is determined. Based on the evaluation question bank and the pre-constructed knowledge graph of the electric power professional field, the second semantic vector corresponding to the knowledge graph can be output through the knowledge graph. Finally, by determining the correlation between the first semantic vector and the second semantic vector, the evaluation of the semantic big model in the electric power professional scenario is completed. The evaluation method provided by the present invention can not only objectively and accurately evaluate the semantic big model in the electric power professional scenario, but also improve the speed of evaluation.

[0113] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0114] The following is an embodiment of the device of the present invention. For details not described in detail, reference may be made to the corresponding method embodiments described above.

[0115] Figure 3 The structural diagram of the evaluation device of the semantic big model in the electric power professional scenario provided by the embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0116] like Figure 3 As shown, the evaluation device 300 of the semantic big model in the electric power professional scenario includes:

[0117] A construction module 310 is used to construct an evaluation question bank of the electric power professional semantic big model to be evaluated;

[0118] A first determination module 320 is used to input the evaluation question bank into the electric power professional semantic big model to obtain a first semantic vector;

[0119] A second determination module 330 is used to determine a second semantic vector corresponding to the evaluation question bank based on the evaluation question bank and a pre-constructed knowledge graph in the electric power professional field;

[0120] The evaluation module 340 is used to evaluate the semantic big model in the power professional scenario based on the correlation between the first semantic vector and the second semantic vector.

[0121] In a possible implementation, the second determination module 330 is used to determine the edge relationships and associated nodes of the evaluation question bank in the pre-constructed knowledge graph of the power professional field;

[0122] Based on the edge relationship and the associated nodes, a second semantic vector corresponding to the evaluation question bank is determined.

[0123] In a possible implementation, the evaluation module 340 is configured to determine a first similarity based on a word similarity between the first semantic vector and the second semantic vector;

[0124] Determining a second similarity based on the short text similarity between the first semantic vector and the second semantic vector;

[0125] Determining a third similarity based on the ROUGE values ​​of the first semantic vector and the second semantic vector;

[0126] Based on the first similarity, the second similarity and the third similarity, the semantic big model in the power professional scenario is evaluated.

[0127] In a possible implementation, the evaluation module 340 is configured to determine a similarity index based on the sum of the first similarity, the second similarity, and the third similarity;

[0128] Based on the similarity index and the preset similarity index evaluation standard, the semantic big model in the power professional scenario is evaluated.

[0129] In a possible implementation, the evaluation module 340 is configured to determine a similarity index based on a product of the first similarity, the second similarity, and the third similarity;

[0130] Based on the similarity index and the preset similarity index evaluation standard, the semantic big model in the power professional scenario is evaluated.

[0131] In a possible implementation, the evaluation module 340 is used to determine a similarity difference based on the similarity index and a preset evaluation threshold;

[0132] Determine the level interval corresponding to the similarity difference value based on the preset similarity index interval in which the similarity difference value is located;

[0133] Based on the scores corresponding to the level intervals, the semantic big model in the power professional scenario is evaluated.

[0134] In a possible implementation, the second determination module 330 is used to construct a knowledge graph in the electric power professional field through a crowdsourcing platform;

[0135] Build an entity synonym forest in the power industry based on the bootstrapping method;

[0136] Link to the knowledge graph in the power professional field based on the entity synonym forest;

[0137] A matching query is performed on the evaluation question bank in the knowledge graph of the electric power professional field to determine the second semantic vector.

[0138] In one possible implementation, the evaluation question bank of the electric power professional semantic big model is obtained based on a survey of various application scenarios in the electric power industry and tested with multiple models.

[0139] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0140] Those of ordinary skill in the art will appreciate that the templates, units, and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0141] If the module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the embodiment of the evaluation method for the adequacy of the distribution network in which each electric vehicle and distributed photovoltaic participate. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium.

[0142] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An evaluation method for a semantic big model in a power professional scenario, characterized in that: include: Construct an evaluation question bank of the semantic large model of the power professional to be evaluated; Inputting the evaluation question bank into the electric power professional semantic big model to obtain a first semantic vector; Based on the evaluation question bank and a pre-constructed knowledge graph in the electric power professional field, determining a second semantic vector corresponding to the evaluation question bank; Based on the correlation between the first semantic vector and the second semantic vector, the semantic big model in the power professional scenario is evaluated.

2. The evaluation method of semantic big model in electric power professional scenario according to claim 1 is characterized in that: The determining, based on the evaluation question bank and the pre-constructed knowledge graph in the electric power professional field, a second semantic vector corresponding to the evaluation question bank comprises: Determine the edge relationships and associated nodes of the evaluation question bank in the pre-constructed knowledge graph of the electric power professional field; Based on the edge relationship and the associated nodes, a second semantic vector corresponding to the evaluation question bank is determined.

3. The evaluation method of semantic big model in electric power professional scenario according to claim 1 is characterized in that: The evaluating the semantic big model in the electric power professional scenario based on the correlation between the first semantic vector and the second semantic vector includes: Determining a first similarity based on word similarities between the first semantic vector and the second semantic vector; Determining a second similarity based on the short text similarity between the first semantic vector and the second semantic vector; Determining a third similarity based on ROUGE values ​​of the first semantic vector and the second semantic vector; Based on the first similarity, the second similarity and the third similarity, the semantic big model in the power professional scenario is evaluated.

4. The evaluation method of semantic big model in electric power professional scenario according to claim 3 is characterized in that: The evaluating the semantic big model in the electric power professional scenario based on the first similarity, the second similarity and the third similarity includes: determining a similarity index based on the sum of the first similarity, the second similarity, and the third similarity; Based on the similarity index and the preset similarity index evaluation standard, the semantic big model in the electric power professional scenario is evaluated.

5. The evaluation method of semantic big model in electric power professional scenario according to claim 3 is characterized in that: The evaluating the semantic big model in the electric power professional scenario based on the first similarity, the second similarity and the third similarity includes: Determining a similarity index based on a product of the first similarity, the second similarity, and the third similarity; Based on the similarity index and the preset similarity index evaluation standard, the semantic big model in the electric power professional scenario is evaluated.

6. The evaluation method of the semantic big model in the electric power professional scenario according to claim 4 or 5 is characterized in that: The evaluating of the semantic big model in the electric power professional scenario based on the similarity index and the preset similarity index evaluation standard includes: Determining a similarity difference based on the similarity index and a preset evaluation threshold; Determining a level interval corresponding to the similarity difference value based on a preset similarity index interval in which the similarity difference value is located; Based on the scores corresponding to the level intervals, the semantic big model in the power professional scenario is evaluated.

7. The evaluation method of semantic big model in electric power professional scenario according to claim 1 is characterized in that: The determining, based on the evaluation question bank and the pre-constructed knowledge graph in the electric power professional field, a second semantic vector corresponding to the evaluation question bank comprises: Constructing a knowledge graph in the power professional field through a crowdsourcing platform; Build an entity synonym forest in the power industry based on the bootstrapping method; Linking to the knowledge graph of the power professional field according to the entity synonym forest; A matching query is performed on the evaluation question bank in the knowledge graph of the electric power professional field to determine the second semantic vector.

8. The evaluation method of the semantic big model in the electric power professional scenario according to any one of claims 1 to 7 is characterized in that: The evaluation question bank of the electric power professional semantic big model is obtained based on a survey of various application scenarios in the electric power industry and tested with multiple models.

9. An evaluation device for a semantic large model in a power professional scenario, characterized in that: include: A construction module is used to construct an evaluation question bank of the electric power professional semantic large model to be evaluated; A first determination module is used to input the evaluation question bank into the electric power professional semantic big model to obtain a first semantic vector; A second determination module is used to determine a second semantic vector corresponding to the evaluation question bank based on the evaluation question bank and a pre-constructed knowledge graph in the electric power professional field; An evaluation module is used to evaluate the semantic big model in the power professional scenario based on the correlation between the first semantic vector and the second semantic vector.

10. The evaluation device for semantic big model in electric power professional scenario according to claim 9, characterized in that: The second determination module is used to determine the edge relationships and associated nodes of the evaluation question bank in the pre-constructed knowledge graph of the electric power professional field; Based on the edge relationship and the associated nodes, a second semantic vector corresponding to the evaluation question bank is determined.

Images