Industrial chain large model testing method

Abstract

The invention discloses an industrial chain large model test method. The method comprises the following steps: listing questions to be asked for an industrial chain large model; checking a problem return intention recognition value by using a tool; comparing whether the intention recognition value accords with expectation or not; inputting a question into the large model page to ask the question; and checking the accuracy of the page answer content. By asking questions related to the industry chain and analyzing the questions through the intention recognition value to give an accurate judgment result, the accuracy of page answer content is improved, a comprehensive and detailed large model for testing the industry chain is provided, and effective testing steps can also help research and development personnel to quickly modify defects and guarantee system stability.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a large-scale model testing method for an industrial chain. Background Art

[0002] With the rapid development of artificial intelligence technology, AI question-answering systems have become an important tool for improving industrial chain efficiency and optimizing resource allocation. From raw material procurement to production and manufacturing, to sales and after-sales service, AI question-answering systems can provide companies with timely and accurate information support, accelerate decision-making processes, and reduce operating costs. To ensure that these systems are used to their maximum effectiveness in actual applications, it is essential to conduct comprehensive and detailed testing on them.

[0003] There is currently no effective solution to the above problems. Summary of the invention

[0004] In view of the above technical problems in the related technology, the present invention proposes a large-scale model testing method for an industrial chain, which can overcome the above shortcomings of the prior art.

[0005] To achieve the above technical objectives, the technical solution of the present invention is implemented as follows: A large-scale industrial chain model testing method comprises the following steps: S1 lists the questions to be asked about the industrial chain model; S2 uses the tool to view the problem and return the intent recognition value; S3 compares whether the intent recognition value meets expectations; The specific process of S3 is as follows: S31 Word segmentation and word slot annotation: The word segmentation module divides the sentence into basic word parts, and then annotates them according to the configured word slots. If the word segmentation result does not meet expectations, it can be adjusted through word segmentation intervention; S32 Template and sample matching: The intent recognition system matches the user input through templates and samples. If the query does not meet expectations, check whether the configuration of the template and sample is correct. If there is an abnormality in the template configuration, refer to the relevant documents for adjustment; S33 Expected error analysis: When the actual result does not match the expected result, an expected error analysis is performed; S34 Model optimization: Optimize the model to improve the accuracy of intent recognition; S4 inputs questions into the large model page to ask questions; S5 Check the accuracy of the page answer content.

[0006] Furthermore, the questions to be asked about the industrial chain big model in step S1 include the definition of the industrial chain, the main links, supply chain management, the application of the big model in the industrial chain and the challenges and opportunities it brings.

[0007] Furthermore, the tool in step S2 is Query intent recognition, and the specific process is as follows: S21 Query understanding: The search engine parses the query entered by the user, including spelling correction, query rewriting, and query expansion; S21 Intent Classification: The parsed query is classified into different intent categories through the classification model; S23 Relevance calculation: Based on the query intent and the content of the document, the relevance score between the query and the document is calculated to determine the document that best meets the user's query intent.

[0008] Furthermore, the method for judging the accuracy of the page answer content includes using Python to check the web page content, evaluating the performance of the natural language processing question-answering system, and evaluating relevant indicators of answer quality.

[0009] Furthermore, Python is used to check the webpage content: by writing a Python script, using the requests library to open the webpage and obtain the page content, and check whether the page content contains specific target content; the specific steps are as follows: S51 imports the requests library and defines web page links; S52 uses the requests.get() function to open the web link and obtain the page content; S53 checks whether the page content contains specific target content, which can be achieved by using the string in operator; S54 outputs the inspection result. If the target content is included, "the page content is correct" is output; otherwise, "the page content is incorrect" is output.

[0010] Furthermore, the performance of the natural language processing question answering system is evaluated by precision, recall, and F1-score; The accuracy calculation formula is: Precision = the number of correct answers predicted / the number of predicted answers; The recall calculation formula is: Recall = number of predicted correct answers / number of actual correct answers; The F1-score calculation formula is: F1 = 2 * (Precision * Recall) / (Precision +Recall).

[0011] Furthermore, relevant indicators for evaluating the quality of answers include the completeness of the answers and the coherence of the answers.

[0012] Beneficial effects of the present invention: The present invention asks questions related to the industrial chain, analyzes the questions through intent recognition values ​​and gives accurate judgment results, thereby improving the accuracy of page answer content and providing a comprehensive and detailed test industrial chain model. Effective testing steps can also help R&D personnel quickly modify defects and ensure system stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 It is a flowchart of the industrial chain large model testing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0016] like Figure 1 As shown, a large industrial chain model testing method according to an embodiment of the present invention includes the following steps: S1 lists the questions to be asked about the industrial chain model; S2 uses the tool to view the problem and return the intent recognition value; S3 compares whether the intent recognition value meets expectations; The specific process of S3 is as follows: S31 Word segmentation and word slot annotation: The word segmentation module divides the sentence into basic word parts, and then annotates them according to the configured word slots. If the word segmentation result does not meet expectations, it can be adjusted through word segmentation intervention; S32 Template and sample matching: The intent recognition system matches the user input through templates and samples. If the query does not meet expectations, check whether the configuration of the template and sample is correct. If there is an abnormality in the template configuration, refer to the relevant documents for adjustment; S33 Expected error analysis: When the actual result does not match the expected result, an expected error analysis is performed; S34 Model optimization: Optimize the model to improve the accuracy of intent recognition; S4 inputs questions into the large model page to ask questions; S5 Check the accuracy of the page answers.

[0017] The questions to be asked about the industrial chain big model in step S1 include the definition of the industrial chain, the main links, supply chain management, the application of the big model in the industrial chain and the challenges and opportunities it brings.

[0018] The tool in step S2 is Query intent recognition, and the specific process is as follows: S21 Query understanding: The search engine parses the query entered by the user, including spelling correction, query rewriting, and query expansion; S21 Intent Classification: The parsed query is classified into different intent categories through the classification model; S23 Relevance calculation: Based on the query intent and the content of the document, the relevance score between the query and the document is calculated to determine the document that best meets the user's query intent.

[0019] The method for judging the accuracy of page answer content includes using Python to check web page content, evaluating the performance of natural language processing question-answering systems, and evaluating relevant indicators of answer quality.

[0020] Use Python to check web page content: Write a Python script, use the requests library to open the web page and obtain the page content, and check whether the page content contains specific target content; the specific steps are as follows: S51 imports the requests library and defines web page links; S52 uses the requests.get() function to open the web link and obtain the page content; S53 checks whether the page content contains specific target content, which can be achieved by using the string in operator; S54 outputs the inspection result. If the target content is included, "the page content is correct" is output; otherwise, "the page content is incorrect" is output.

[0021] The performance of the natural language processing question answering system is evaluated by precision, recall and F1-score; The accuracy calculation formula is: Precision = the number of correct answers predicted / the number of predicted answers; The recall calculation formula is: Recall = number of predicted correct answers / number of actual correct answers; The F1-score calculation formula is: F1 = 2 * (Precision * Recall) / (Precision +Recall).

[0022] Relevant indicators for evaluating the quality of responses include the completeness of the responses and the coherence of the responses.

[0023] In order to facilitate understanding of the above technical solutions of the present invention, the above technical solutions of the present invention are described in detail below through specific usage methods.

[0024] In specific use, the large-scale model testing method of the industrial chain according to the present invention includes the following steps: 1. List the questions about the industrial chain model; For example: What is the **industry chain like? What is the **industry chain map like? **What is the overall layout of the industrial chain? / Please generate an industrial map of the **industry chain? In the second part, what enterprises are there in City A? How many relevant patents does the company have in the industry chain? In which links are these patents concentrated? (Question rephrased: In which links are the patents applied for by **enterprises in the *industry chain concentrated?) In which links of the **industry chain do enterprises have invention patents? **What is the development trend of patent applications of enterprises in *fields? **In which links of the **industry chain do enterprises have layout? Development trend of the industry chain* In which links of the ** industrial chain does City A have a layout? In which links of the **industry chain does City A have advantages? What are the weak links in the *industry chain of City A? Which foreign enterprises can be introduced? What are the differences between A and B in the layout of the ** industrial chain? Please compare the comprehensive strength of places A and B in the **link of the **industrial chain.

[0025] **What are the high-end links in the industry chain? Which high-end links of the **industry chain has City A deployed? **What are the leading enterprises in this link? / **What are the leading enterprises in this field? **What are the regions with advantages in the link? / ** In which regions are enterprises in the field mainly distributed? 2. Use the tool to view the problem return intent recognition value; Query intent recognition is a search engine that analyzes the query (Query) entered by the user, understands and predicts the user's query intent, and returns the most relevant and useful search results. This technology involves an in-depth understanding of the query, the understanding of the content (documents), and the matching and sorting between the query and the content.

[0026] 2. Technical Principle The technical principles of query intent recognition mainly include the following aspects: Query understanding: The search engine first needs to parse the query entered by the user, including spelling correction, query rewriting, query expansion and other steps to improve the accuracy and relevance of the query.

[0027] Intent classification: The parsed query is classified into different intent categories through a classification model. This process is usually based on machine learning algorithms such as Bayesian classifiers, support vector machines (SVMs), neural networks, etc.

[0028] Relevance calculation: Based on the query intent and the content of the document, the relevance score between the query and the document is calculated to determine which documents best meet the user's query intent.

[0029] 3. Compare the intent recognition values ​​to see if they meet expectations; Whether the intent recognition value meets expectations can be compared and evaluated through the following steps: Word segmentation and word slot annotation: First, the sentence will be divided into basic word parts by the word segmentation module, and then annotated according to the configured word slots. If the word segmentation result does not meet expectations, it can be adjusted through word segmentation intervention, such as separating two words that are combined together, or adding word slots from a custom dictionary.

[0030] Template and sample matching: The intent recognition system matches the user input through templates and samples. If the query does not meet expectations, you need to check whether the template and sample configurations are correct. If there is an abnormality in the template configuration, you can refer to the relevant documents for adjustment.

[0031] Expected error analysis: When the actual result does not match the expected result, an expected error analysis is required. The expected error hypothesis holds that the degree of concern of individuals for key events in a situation will affect the setting of expected values. The higher the degree of concern, the greater the expected error, and the more people tend to believe that the result is intentional by the actor.

[0032] Model optimization: To improve the accuracy of intent recognition, the model can be optimized. For example, a pre-trained model such as BERT can be used for fine-tuning, or the Prompt technology can be used to reduce the need for training data3.

[0033] Through the above steps, we can effectively compare and evaluate whether the intent recognition value meets expectations, and take corresponding measures to optimize it.

[0034] 4. Enter questions on the large model page to ask questions; 5. Check the accuracy of the answers on the page; There are several ways to check the accuracy of the answers on a page: Use Python to check web page content: You can write a Python script, use the requests library to open the web page and obtain the page content, and then check whether the page content contains specific target content. The specific steps are as follows: Import the requests library and define the web link.

[0035] Use the requests.get() function to open a web link and get the page content.

[0036] Checking whether the page content contains specific target content can be achieved through the string in operator.

[0037] Output the check result. If the target content is included, output "Page content is correct", otherwise output "Page content is incorrect".

[0038] Evaluating the performance of natural language processing question answering systems: For natural language processing (NLP) question answering systems, the accuracy of answers can be evaluated by the following metrics: Precision: The proportion of correct answers among the results predicted as correct answers. The calculation formula is: Precision = Number of correct answers predicted / Number of predicted answers.

[0039] Recall: The proportion of correct answers that are predicted correctly among the actual correct answers. The calculation formula is: Recall = Number of predicted correct answers / Number of actual correct answers.

[0040] F1-score: The harmonic mean of precision and recall, which comprehensively reflects the accuracy of the system. The calculation formula is: F1 = 2 * (Precision * Recall) / (Precision + Recall).

[0041] Relevant indicators for evaluating the quality of answers: Completeness of answer: Human judgment on whether the answer covers all the information required by the question.

[0042] Coherence of answer: Assess the logic and organization of the answer.

[0043] To sum up, with the help of the above-mentioned technical solution of the present invention, by asking questions related to the industrial chain and analyzing the questions through intent recognition values ​​to give accurate judgment results, the accuracy of the page answer content is improved, and a comprehensive and detailed test industry chain model is provided. Effective testing steps can also help R&D personnel quickly modify defects and ensure system stability.

[0044] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A large-scale industrial chain model testing method, characterized in that: The steps include: S1 lists the questions to be asked about the industrial chain model; S2 uses the tool to view the problem and return the intent recognition value; S3 compares whether the intent recognition value meets expectations; The specific process of S3 is as follows: S31 Word segmentation and word slot annotation: The word segmentation module divides the sentence into basic word parts, and then annotates them according to the configured word slots. If the word segmentation result does not meet expectations, it can be adjusted through word segmentation intervention; S32 Template and sample matching: The intent recognition system matches the user input through templates and samples. If the query does not meet expectations, check whether the configuration of the template and sample is correct. If there is an abnormality in the template configuration, refer to the relevant documents for adjustment; S33 Expected error analysis: When the actual result does not match the expected result, an expected error analysis is performed; S34 Model optimization: Optimize the model to improve the accuracy of intent recognition; S4 inputs questions into the large model page to ask questions; S5 Check the accuracy of the page answers.

2. The large-scale model testing method of the industrial chain according to claim 1 is characterized in that: The questions to be asked about the industrial chain big model in step S1 include the definition of the industrial chain, the main links, supply chain management, the application of the big model in the industrial chain and the challenges and opportunities it brings.

3. The large-scale model testing method of the industrial chain according to claim 1 is characterized in that: The tool in step S2 is Query intent recognition, and the specific process is as follows: S21 Query understanding: The search engine parses the query entered by the user, including spelling correction, query rewriting, and query expansion; S21 Intent Classification: The parsed query is classified into different intent categories through the classification model; S23 Relevance calculation: Based on the query intent and the content of the document, the relevance score between the query and the document is calculated to determine the document that best meets the user's query intent.

4. The large-scale model testing method of the industrial chain according to claim 1 is characterized in that: The method for judging the accuracy of page answer content includes using Python to check web page content, evaluating the performance of natural language processing question-answering systems, and evaluating relevant indicators of answer quality.

5. The large-scale model testing method of the industrial chain according to claim 4 is characterized in that: Use Python to check web page content: Write a Python script, use the requests library to open the web page and obtain the page content, and check whether the page content contains specific target content; the specific steps are as follows: S51 imports the requests library and defines web page links; S52 uses the requests.get() function to open the web link and obtain the page content; S53 checks whether the page content contains specific target content, which can be achieved by using the string in operator; S54 outputs the inspection result. If the target content is included, "the page content is correct" is output; otherwise, "the page content is incorrect" is output.

6. The large-scale model testing method for the industrial chain according to claim 4 is characterized in that: The performance of the natural language processing question answering system is evaluated by precision, recall and F1-score; The accuracy calculation formula is: Precision = the number of correct answers predicted / the number of predicted answers; The recall calculation formula is: Recall = number of predicted correct answers / number of actual correct answers; The F1-score calculation formula is: F1 = 2 * (Precision * Recall) / (Precision + Recall).

7. The large-scale model testing method of the industrial chain according to claim 4 is characterized in that: Relevant indicators for evaluating the quality of responses include the completeness of the responses and the coherence of the responses.

Images