Enterprise artificial intelligence development level measurement method based on large language model method

Abstract

The invention discloses an enterprise artificial intelligence development level measurement method based on a large language model method. The method comprises the steps of modeling data acquisition, data preprocessing, to-be-predicted sentence library labeling, large model supervision and fine adjustment, model effectiveness evaluation and verification and artificial intelligence index construction. According to the method, the development level of artificial intelligence is measured from the perspective of enterprises, and the perspective of artificial intelligence measurement is greatly enriched; based on enterprise annual report data, an industrial chain upstream and downstream index system covering a base layer, a technology layer and an application layer is designed, and powerful support is provided for accurately evaluating the artificial intelligence development level of an enterprise. The large language model is introduced for text analysis, the large language model is applied to measurement of the artificial intelligence development level for the first time, the data collection and processing efficiency is improved, high automation is achieved, meanwhile, the credibility and repeatability of results are ensured, and a brand-new methodology path is developed for research in related fields.

Description

Technical Field

[0001] The present invention relates to a method for measuring the development level of artificial intelligence, in particular to a method for measuring the development level of artificial intelligence of an enterprise based on a large language model method, and belongs to the technical field of artificial intelligence technology evaluation. Background Art

[0002] With the advent of the digital, networked, and intelligent era, artificial intelligence (AI) technology has been widely used in various fields, including corporate management, healthcare, finance, education, etc. However, how to scientifically and systematically evaluate the level of application of artificial intelligence technology in enterprises has become an urgent problem to be solved. In exploring the measurement method of the development level of artificial intelligence in enterprises, some literature has made preliminary attempts. Dauth and Findeisen (2017) quantified the degree of application by analyzing the frequency of artificial intelligence-related words in corporate annual reports, while Fujii and Managi (2018) used questionnaire surveys to directly understand the use of artificial intelligence in enterprises in operations. However, the questionnaire survey method is susceptible to the halo effect (Yao Jiaquan et al., 2024), resulting in uneven data quality, large sample limitations, poor comprehensiveness, and high time and labor costs, which limits the reliability and repeatability of this method. Wang Yongqin and Dong Wen (2020) tried to use robot penetration rate as an evaluation indicator, which has certain reference value, but there is still room for improvement in accuracy. Other studies have adopted the dictionary method to construct indicators based on the annual report text of listed companies, but the credibility of the indicators is controversial due to the lack of accuracy of the existing dictionary in the classification of professional terms. To improve this situation, Yao Jiaquan et al. (2024) introduced supervised machine learning technology to construct an AI-specific dictionary by identifying similar words, striving to fully cover relevant professional vocabulary, and construct enterprise AI development indicators based on this, effectively improving the credibility of the measurement. Despite this, this type of method is essentially a variant of the bag-of-words model, ignoring the word order, grammatical structure and context association of the text (Shen Yan et al., 2019), thus forming an "impossible triangle" conflict between the accuracy, cost control and repeatability of text analysis (Li Chuntao et al., 2024). Therefore, to accurately measure the development level of enterprise AI, the key lies in breaking through the limitations of manual discrimination, bag-of-words model and supervised machine learning, and exploring a new measurement method that can comprehensively consider the deep characteristics of the text, cost control and reproducibility of the results, so as to optimize and promote the research process of measuring the development level of enterprise AI.

[0003] The research on the development level of artificial intelligence has achieved an evolution from a single dimension to multiple dimensions in the construction of indicators, and the data foundation has gradually expanded from structured data to unstructured data. This transformation has effectively promoted the in-depth study of the progress of artificial intelligence itself and its application status. However, there are still some areas for improvement in current research. First, most of the existing research is based on a macro perspective (Wang Linhui et al., 2022). Although this is of great help in grasping the regional and industry profiles, it ignores the detailed examination of the development level of artificial intelligence of individual enterprises, hindering the in-depth study of the research to a more micro level. Secondly, the existing indicator system mainly constructs enterprise artificial intelligence-related dictionaries based on a technical perspective, lacks a comprehensive evaluation from the upstream and downstream perspectives of the industrial chain, and is difficult to fully reflect the comprehensive development status of enterprises in the field of artificial intelligence, such as infrastructure, technology development, and application scenarios. Finally, when using the text of corporate annual reports to construct indicators, there are severe challenges between credibility, cost and repeatability. The latest research points out that the large model pre-trained generative language model GPT (Generative Pre-trained Transformer) provides a new idea for effectively solving the application difficulties of text information in the construction of economic indicators (Li Chuntao, 2024). Based on this, this application proposes a method for measuring the level of corporate artificial intelligence development based on a large language model method. Summary of the invention

[0004] The purpose of the present invention is to provide a method for measuring the development level of artificial intelligence in enterprises based on a large language model method in order to solve at least one of the above technical problems. By scientifically and systematically evaluating the development level of artificial intelligence technology in enterprises, the present invention helps enterprises better understand their own advantages and disadvantages in artificial intelligence technology, and then guides enterprises to formulate targeted artificial intelligence technology development strategies, and enhance the overall competitiveness and intelligence level of enterprises. At the same time, it provides a basis for the formulation of policies related to the development of artificial intelligence.

[0005] The present invention achieves the above-mentioned purpose through the following technical scheme: A method for measuring the development level of artificial intelligence of an enterprise based on a large language model method comprises the following steps:

[0006] S1. Obtain modeling data by combining crawler technology with manual collection to fully obtain the full text of listed companies' annual reports as the data source for modeling;

[0007] S2, data preprocessing, processing the collected annual report texts, dividing them into independent sentences, and building the sentence library foundation to be predicted;

[0008] S3. Label the sentence library to be predicted, use machine labeling technology to build a "reserved word library" to efficiently and accurately screen out sentences closely related to artificial intelligence;

[0009] S4: supervised fine-tuning of the large model, deep learning and supervised fine-tuning of the extracted sentences, and accurate determination of the category label of each sentence through sentence-by-sentence reasoning and prediction;

[0010] S5. Model effectiveness evaluation and verification, using multiple verification methods such as confusion matrix analysis and comparison with authoritative public reports to comprehensively evaluate model performance and ensure the scientificity and rationality of the measurement method;

[0011] S6. Construction of artificial intelligence indicators. By adding 1 to the number of artificial intelligence-related sentences in each dimension and taking the natural logarithm, we construct the enterprise artificial intelligence development index to provide a quantitative reference for the enterprise's intelligent transformation.

[0012] As a further solution of the present invention: in S1, the contents of the manually collected annual reports of listed companies cover the financial status, operating results and necessary disclosure information of the company throughout the entire fiscal year, and the obtained annual reports of listed companies include annual reports of all years since the implementation of the new accounting standards.

[0013] As a further solution of the present invention: In S1, annual reports of each company's official website are crawled by crawler technology, and combined with manual search, to comprehensively collect relevant annual report information.

[0014] As a further solution of the present invention: In S3, the constructed "reserved word library" covers the following four aspects:

[0015] Taking into full consideration the practical perspective of the business community, and starting from the perspective of the upstream and downstream of the industrial chain, we divide AI companies into four major sectors: non-AI, basic layer, technology layer, and application layer. We also carefully sort out the sub-indicators at each level to reflect the actual operation and development trends of the industry.

[0016] In-depth research on policy orientation, clarify the structural framework of the AI ​​standard system, as well as the key elements of technology, products, platforms, and industry applications covered therein, and further refine and incorporate more specific key terms based on the non-AI, basic layer, technical layer, and application layer;

[0017] Drawing on the research results of the academic community and referring to the conclusions of existing materials on artificial intelligence keyword selection, we can further enrich the dictionary content;

[0018] It gives full play to the fine optimization role of manual judgment. Based on a large number of research reports, policy texts, academic documents and corporate annual reports, it continuously analyzes, refines and marks key words through manual reading, and finally forms a dictionary of core keywords of artificial intelligence.

[0019] As a further solution of the present invention: In S4, deep learning of the extracted sentences specifically includes:

[0020] Construct a "reserved word library" to accurately eliminate sentences that are not related to artificial intelligence. The "reserved word library" strategy ensures that sentences closely related to "artificial intelligence" are retained as much as possible. If a sentence contains any word in the reserved word library, it is judged to be related to "artificial intelligence"; otherwise, it is judged to be irrelevant in the annual report text processing stage;

[0021] Sentences are split according to punctuation marks such as commas, periods, semicolons, colons, question marks, exclamation marks, and spaces. The split sentence data is deeply analyzed through the "retained vocabulary" to ensure efficient and accurate determination of whether the sentence data is related to "artificial intelligence".

[0022] As a further solution of the present invention: In S4, the supervised fine-tuning of the extracted sentences specifically includes:

[0023] By using the BERT model to perform unsupervised learning on a large-scale text corpus, the BERT model has bidirectional encoding and understanding capabilities. The BERT model only needs simple fine-tuning to quickly adapt to specific task requirements.

[0024] As a further solution of the present invention: in S5, the confusion matrix analysis specifically includes:

[0025] The prediction results and actual conditions are evaluated through the confusion matrix classification model, and multiple evaluation indicators such as accuracy, precision, recall and F1 score are calculated accordingly; a four-category confusion matrix is ​​generated through the model and normalized by column to intuitively analyze the recall and precision of each category.

[0026] The beneficial effects of the present invention are:

[0027] This invention expands the measurement of artificial intelligence from the micro perspective of enterprises, measures the development level of artificial intelligence from the micro perspective of enterprises, and greatly enriches the perspective of artificial intelligence measurement; constructs an enterprise artificial intelligence indicator system covering the industrial chain. Relying on the annual report data of enterprises, an indicator system covering the upstream and downstream of the industrial chain at the basic layer, technical layer, and application layer is designed, which provides strong support for accurately evaluating the artificial intelligence development level of enterprises, so as to comprehensively reflect the layout and strength of enterprises in the field of artificial intelligence;

[0028] This invention introduces a large language model for text analysis and applies the large language model to the measurement of the level of artificial intelligence development for the first time. It not only significantly improves the efficiency of data collection and processing, achieves a high degree of automation, but also effectively reduces the measurement cost. While ensuring the credibility and repeatability of the results, it opens up a new methodological path for research in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1This is a schematic diagram of the enterprise artificial intelligence measurement process of the present invention;

[0030] Figure 2 A schematic diagram of the enterprise artificial intelligence dictionary of the present invention;

[0031] Figure 3 Schematic diagram of the confusion matrix of the artificial intelligence measurement model of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Embodiment 1

[0034] like Figure 1 As shown, a method for measuring the development level of artificial intelligence of an enterprise based on a large language model method includes the following steps:

[0035] S1. Obtain modeling data. Through a combination of crawler technology and manual collection, the full text of the annual reports of listed companies is fully obtained as the data source for modeling. In the previous text analysis technology for determining data sources, the word frequency method is used for text analysis, which often focuses on the MD&A (Management Discussion and Analysis) and "Table of Contents, Interpretations and Major Risk Warnings" sections in the annual report. This method of focusing on specific chapters can effectively eliminate the noise interference of non-core content in the annual report and improve the pertinence of word frequency analysis, but it may also introduce the risk of sampling errors, that is, it will miss important information contained in other parts, thereby affecting the comprehensiveness and accuracy of data analysis. Therefore, while pursuing the accuracy of analysis, it is also necessary to weigh the possible impact of the sampling range on the results to ensure the reliability and objectivity of the final conclusion. Due to the significant progress of large language models in semantic understanding, the selection of annual report full-text data for analysis can effectively reduce information confusion and interference and grasp the text intent more accurately;

[0036] S2, data preprocessing, processing the collected annual report texts, dividing them into independent sentences, and building the sentence library foundation to be predicted;

[0037] S3. Label the sentence library to be predicted. Use machine labeling technology to build a "reserved word library" to efficiently and accurately screen out sentences closely related to artificial intelligence. Compared with the traditional "direct recognition method", this strategy is simpler and more efficient in operation, and its accuracy is significantly improved. At the same time, in order to ensure the representativeness and balance of the sample, the sentence library to be predicted is scientifically sampled, and 1.0% of the annual report of each company is randomly sampled each year; then, these sentence samples are compared with the enterprise artificial intelligence dictionary in the four sub-sectors of "non-artificial intelligence", "basic layer", "technical layer" and "application layer" (see Table 1 and Figure 2 ) to manually label and classify, laying a solid foundation for subsequent model training and testing;

[0038]

[0039] S4. Supervised fine-tuning of large models: deep learning and supervised fine-tuning of extracted sentences. Through sentence-by-sentence reasoning and prediction, the category label of each sentence is accurately determined. Previous text analysis technology was based on keyword methods and could not identify the meaning of the context and the potential meaning of the sentence, which easily led to inaccurate measurements. The BERT model used is a classic model in the field of natural language processing. It can have two-way encoding and understanding capabilities through unsupervised learning on a large-scale text corpus. This feature allows the BERT model to quickly adapt to specific task requirements with only simple fine-tuning. Therefore, combined with the above-mentioned pre-training and supervised fine-tuning strategies, as well as the powerful capabilities of the BERT model, a large language model for text analysis for enterprise artificial intelligence development is effectively constructed and optimized;

[0040] S5. Model effectiveness evaluation and verification, using confusion matrix analysis and multiple verification methods such as comparison with authoritative public reports to comprehensively evaluate model performance and ensure the scientificity and rationality of the measurement method;

[0041] S6. Construction of artificial intelligence indicators. By adding 1 to the number of artificial intelligence-related sentences in each dimension and taking the natural logarithm, we construct the enterprise artificial intelligence development index to provide a quantitative reference for the enterprise's intelligent transformation.

[0042] Embodiment 2

[0043] In addition to all the technical features of the first embodiment, this embodiment also includes:

[0044] The contents of the annual reports of listed companies collected manually cover the company's financial status, operating results and necessary disclosure information for the entire fiscal year, and the annual reports of listed companies obtained include annual reports of all years since the implementation of the new accounting standards. According to the regulations of the China Securities Regulatory Commission, listed companies must publish their financial reports of the previous year to the stock exchange and investors every year to show the company's financial health and operating performance. In order to attract market attention and demonstrate their technological innovation capabilities, listed companies often disclose their progress and achievements in the development and application of artificial intelligence technology in detail in their annual reports, and use the annual reports of all A-share listed companies as text analysis objects for measuring the level of corporate artificial intelligence development. Considering the potential adverse effects of changes in accounting standards on data consistency, annual report data since the implementation of the new accounting standards in 2007 are specially selected for analysis, with a time span from 2007 to 2020, to ensure the standardization and comparability of research data, and provide scientific data support for accurately evaluating the development of corporate artificial intelligence.

[0045] Furthermore, in order to deeply analyze the annual reports of listed companies, the present invention has comprehensively collected relevant annual report materials by crawling the official websites of various companies. In the process of using the word frequency method for text analysis, it is different from other artificial intelligence measurement technologies that focus on the MD&A (Management Discussion and Analysis) and "Table of Contents, Interpretations and Major Risk Warnings" sections in the annual report. The original measurement technology's specific chapter method can effectively eliminate the noise interference of non-core content in the annual report and improve the pertinence of word frequency analysis, but it may also introduce the risk of sampling errors, that is, it will miss important information contained in other parts, thereby affecting the comprehensiveness and accuracy of data analysis. Therefore, while pursuing the accuracy of analysis, it is also necessary to weigh the possible impact of the sampling range on the results to ensure the reliability and objectivity of the final conclusion. The significant progress of large language models in semantic understanding can effectively reduce information confusion and interference and grasp the text intent more accurately. Through analysis, 37,584 annual reports of 4,181 companies from 2007 to 2020 were collected, as shown in the following table; compared with the actual situation, the total number of A-share listed companies in 2020 was 4,287, while in 2007 it was 1,530. This data is highly consistent with the year distribution of the selected sample, which strongly proves the representativeness of the sample. In addition, it is worth noting that the proportion of artificial intelligence companies in the sample has increased significantly from 30.3% in 2007 to 92.9% in 2020. This trend clearly reveals the year-on-year increase in the application rate of artificial intelligence in listed companies.

[0046]

[0047] When constructing the AI ​​dictionary, the industry, policy and academic perspectives were comprehensively considered, and the dictionary was optimized through manual reading to ensure the comprehensiveness, accuracy and practicality of the dictionary. Specifically, the construction method covers the following four aspects: First, the practical perspective of the business community was fully considered. Referring to the classification methods of authoritative institutions such as Deloitte and Prospective Industry Research Institute, AI companies were divided into three major sectors: basic layer, technical layer and application layer from the perspective of the upstream and downstream of the industrial chain, and the subdivision indicators at each level were carefully sorted out to reflect the actual operation and development trend of the industry. Secondly, the policy orientation was studied in depth. Based on important policy documents such as the "Guidelines for the Construction of the National New Generation Artificial Intelligence Standard System" and the "New Generation Artificial Intelligence Development Plan", the structural framework of the AI ​​standard system and the key elements such as technology, products, platforms, and industry applications covered therein were clarified. These policy documents not only provide authoritative definitions of terms, but also guide the further refinement and inclusion of more specific key words based on the basic layer, technical layer and application layer. Furthermore, the research results of the academic community were drawn upon. In particular, the conclusions of Yao Jiaquan et al. (2024) on the selection of AI keywords were referred to further enrich the content of the dictionary. Finally, the fine optimization effect of human judgment is fully utilized. Based on a large number of research reports, policy texts, academic documents, corporate annual reports and other diversified materials, through manual reading, key words are continuously analyzed, refined and marked, and finally a dictionary containing 152 core keywords of artificial intelligence is formed.

[0048] like Figure 2 As shown in the figure, during the annual report text processing stage, sentences are split according to punctuation marks such as commas, periods, semicolons, colons, question marks, exclamation marks, and spaces. Through in-depth analysis of the split sentence data, it is observed that most sentences have no direct connection with "artificial intelligence". Faced with this serious imbalance in the number of positive and negative samples, if the fine-tuned Bert-Base-Chinese model is directly applied for sentence classification, not only will there be problems of low training efficiency and increased inference latency, but also the classification accuracy is difficult to guarantee. To this end, an innovative "reserved word library" is constructed to accurately eliminate sentences that are not related to artificial intelligence. Compared with directly identifying related sentences, our "reserved word library" strategy ensures that sentences closely related to artificial intelligence are retained as much as possible, which is simpler in operation and can significantly improve classification accuracy. In actual applications, if a sentence contains any word in the reserved word library, we determine that it is related to "artificial intelligence"; otherwise, it is determined to be irrelevant.

[0049] like Figure 3As shown in the figure, after the inference is completed, the confusion matrix is ​​used to evaluate the model. As a key classification model evaluation tool, the confusion matrix can scientifically compare the prediction results of the model with the actual situation, and calculate multiple evaluation indicators such as accuracy, precision, recall and F1 score. A confusion matrix of four categories is generated by the model, and normalized by column to intuitively analyze the recall and precision of each category. Except for the technical layer, the values ​​on the diagonal are all above 0.5, specifically 0.97, 0.84 and 0.77, which fully demonstrates the excellent performance of the model in the classification task. Further, we calculated the various evaluation indicators of the model based on the confusion matrix and summarized them in the table below. From the results, the accuracy of all categories exceeds 0.9, which means that the proportion of samples predicted correctly by the model accounts for more than 90% of the total number of samples. In terms of accuracy, except for the technical layer, the accuracy of other categories exceeds 50%, that is, the model is accurate at least 50 times out of every 100 predictions for a certain category (non-technical layer). In addition, in addition to the technical layer, the recall rate of each category is also maintained at around 70%, further verifying the high efficiency of the model in identifying relevant categories.

[0050]

[0051]

[0052] Working principle: Through a combination of crawler technology and manual collection, the full text of the annual reports of listed companies is fully obtained as the data source for modeling; the collected annual report texts are processed and divided into independent sentences to build the basis of the sentence library to be predicted; machine annotation technology is used to build a "reserved vocabulary" to efficiently and accurately screen out sentences closely related to artificial intelligence; deep learning and supervised fine-tuning are carried out on the extracted sentences, and the category label of each sentence is accurately determined through sentence-by-sentence reasoning and prediction; confusion matrix analysis and comparison with authoritative public reports are used for multiple verification methods to comprehensively evaluate the model performance and ensure the scientificity and rationality of the measurement method; by adding 1 to the number of artificial intelligence-related sentences in each dimension and taking the natural logarithm, an enterprise artificial intelligence development indicator is constructed to provide a quantitative reference for the intelligent transformation of enterprises.

[0053] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0054] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for measuring the development level of artificial intelligence of an enterprise based on a large language model method, characterized in that: The following steps are involved: S1. Obtain modeling data by combining crawler technology with manual collection to fully obtain the full text of listed companies' annual reports as the data source for modeling; S2, data preprocessing, processing the collected annual report texts, dividing them into independent sentences, and building the sentence library foundation to be predicted; S3. Label the sentence library to be predicted, use machine labeling technology to build a "reserved word library" to efficiently and accurately screen out sentences closely related to artificial intelligence; S4: supervised fine-tuning of the large model, deep learning and supervised fine-tuning of the extracted sentences, and accurate determination of the category label of each sentence through sentence-by-sentence reasoning and prediction; S5. Model effectiveness evaluation and verification, using confusion matrix analysis and multiple verification methods such as comparison with authoritative public reports to comprehensively evaluate model performance and ensure the scientificity and rationality of the measurement method; S6. Construction of artificial intelligence indicators. By adding 1 to the number of artificial intelligence-related sentences in each dimension and taking the natural logarithm, we construct the enterprise artificial intelligence development index to provide a quantitative reference for the enterprise's intelligent transformation.

2. The method for measuring the development level of artificial intelligence of an enterprise based on a large language model method according to claim 1 is characterized by: In S1, the contents of the manually collected annual reports of listed companies cover the company's financial status, operating results and necessary disclosure information for the entire fiscal year, and the annual reports of listed companies obtained include annual reports of all years since the implementation of the new accounting standards.

3. The method for measuring the development level of artificial intelligence of an enterprise based on a large language model method according to claim 1 is characterized by: In S1, the annual reports of the official websites of various companies are crawled by using crawler technology, and combined with manual search methods to comprehensively collect relevant annual report information.

4. The method for measuring the development level of artificial intelligence of an enterprise based on a large language model method according to claim 1 is characterized in that: In the S3, the "reserved word library" constructed covers the following four aspects: Taking into full consideration the practical perspective of the business community, and starting from the perspective of the upstream and downstream of the industrial chain, we divide AI companies into four major sectors: non-AI, basic layer, technology layer, and application layer. We also carefully sort out the sub-indicators at each level to reflect the actual operation and development trends of the industry. In-depth research on policy orientation, clarify the structural framework of the AI ​​standard system, as well as the key elements of technology, products, platforms, and industry applications covered therein, and further refine and incorporate more specific key terms based on the non-AI, basic layer, technical layer, and application layer; Drawing on the research results of the academic community and referring to the conclusions of existing materials on artificial intelligence keyword selection, we can further enrich the dictionary content; It gives full play to the fine optimization role of manual judgment. Based on a large number of research reports, policy texts, academic documents and corporate annual reports, it continuously analyzes, refines and marks key words through manual reading, and finally forms a dictionary of core keywords of artificial intelligence.

5. The method for measuring the development level of artificial intelligence of an enterprise based on a large language model method according to claim 1 is characterized by: In S4, deep learning of the extracted sentences specifically includes: Build a "reserved word library" to accurately eliminate sentences that are not related to artificial intelligence. The "reserved word library" strategy ensures that sentences that are closely related to artificial intelligence are retained as much as possible. If a sentence contains any word in the reserved word library, it is judged to be related to "artificial intelligence"; otherwise, it is judged to be irrelevant; During the annual report text processing stage, sentences are split according to commas, periods, semicolons, colons, question marks, exclamation marks and space punctuation marks; the split sentence data is deeply analyzed through the "reserved vocabulary" to ensure efficient and accurate determination of whether the sentence data is related to "artificial intelligence".

6. The method for measuring the development level of artificial intelligence of an enterprise based on a large language model method according to claim 5 is characterized by: In S4, the supervised fine-tuning of the extracted sentences specifically includes: By using the BERT model for unsupervised learning on a large-scale text corpus, the BERT model has bidirectional encoding and understanding capabilities. The BERT model only needs simple fine-tuning to quickly adapt to specific task requirements.

7. The method for measuring the development level of artificial intelligence of an enterprise based on a large language model method according to claim 1 is characterized in that: In S5, the confusion matrix analysis specifically includes: The prediction results and actual conditions are evaluated through the confusion matrix classification model, and multiple evaluation indicators such as accuracy, precision, recall and F1 score are calculated accordingly; a four-category confusion matrix is ​​generated through the model and normalized by column to intuitively analyze the recall and precision of each category.

Images