Script classification method and device and electronic equipment

Abstract

The embodiment of the invention discloses a script classification method and device and electronic equipment, and the method comprises the steps: constructing a script knowledge base, a script intention base and a script category base according to an internal document and a script data set; according to the to-be-classified scripts, related documents are obtained from the script knowledge base, related intentions are obtained from the script intention base, and related candidate classes are obtained from the class base; constructing a thinking chain according to the related document, the related intention and the candidate class; inputting the thinking chain into a target large language model for analysis to obtain a target category corresponding to the to-be-classified script; according to the method, intelligent classification is carried out on the scripts generated by operation and maintenance in an enterprise, part of candidate classes are given when the scripts are input, meanwhile, automatic intention description generation is carried out on the input scripts, and the intention description associated with the input scripts is recognized by utilizing the inference ability and related documents of a large language model; while the script classification efficiency is improved, the script classification accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of production operation and maintenance technology; specifically, to a script classification method, device and electronic equipment. Background Art

[0002] Scripts usually contain multiple parts such as scene commands, parameters, function definitions and explanations. In the production and operation and maintenance field of Internet companies, script classification is a key link in standardized processing and plays a vital role in ensuring code security and permission privacy.

[0003] Traditional script classification is mainly based on manual labeling, which relies on the staff's manual experience to label and classify scripts. This not only makes script classification costly and inefficient, but also has the problem of subjectivity and instability in classification labeling, which affects the accuracy of script classification management.

[0004] It can be seen that how to improve the efficiency and accuracy of script classification is a problem that needs to be solved urgently. Summary of the invention

[0005] In order to solve the above technical problems, the embodiments of the present application provide a script classification method, device and electronic device, which solve the problems of low efficiency and low accuracy in script classification in related technologies.

[0006] According to one aspect of an embodiment of the present application, a script classification method is provided, the method comprising: constructing a script knowledge base, a script intent library and a script category library based on internal documents and script data sets; obtaining relevant documents from the script knowledge base, obtaining relevant intents from the script intent library and obtaining relevant candidate classes from the category library based on the script to be classified; constructing a thought chain based on the relevant documents, the relevant intents and the candidate classes; inputting the thought chain into a target large language model for analysis to obtain a target category corresponding to the script to be classified.

[0007] Optionally, when the script data set includes a closed-source script data set and an open-source script data set, constructing a script intent library based on the script data set includes: obtaining a first statistical feature of the closed-source script data set and a second statistical feature of the open-source script data set, the statistical features including script length, script language and commonly used commands of the script; obtaining script data related to the closed-source script data set from the open-source script data set based on the distribution difference between the first statistical feature and the second statistical feature to obtain an external script data set; obtaining a matching intent recognition model based on the first statistical feature; performing intent recognition on the closed-source script data set and the external script data set respectively according to the intent recognition model to obtain internal commonly used command intent pairs and external general command intent pairs; and constructing the script intent library based on the internal commonly used command intent pairs and the external general command intent pairs.

[0008] Optionally, when there is a special command with an empty intent in the internal commonly used command intention pair, the method further includes: preprocessing the special command to obtain a target command; acquiring a target intent and an Internet document corresponding to the target command based on an Internet search, and inserting the target intent into the script intent library and updating the Internet document to the internal knowledge base.

[0009] Optionally, according to the script to be classified, relevant documents are obtained from the script knowledge base, relevant intentions are obtained from the script intention library, and relevant candidate classes are obtained from the category library, including: vectorizing the script knowledge base, the script intention library and the script category library to obtain a vector cluster corresponding to each script category; according to the center distance from the target vector corresponding to the script to be classified to each vector cluster, at least two candidate vector clusters matching the target vector are obtained; and relevant documents, relevant intentions and relevant candidate classes corresponding to the script to be classified are obtained from the at least two candidate vector clusters.

[0010] Optionally, the script knowledge base, the script intent library and the script category library are vectorized to obtain vector clusters corresponding to each script category, including: performing vector extraction on the data in the script knowledge base, the script intent library and the script category library to obtain vector data sets corresponding to different categories; selecting K initial cluster centers according to the mean and standard deviation of the vector data sets; calculating the distance from each vector data to each cluster, and dividing each vector data into clusters with the minimum distance; calculating new cluster centers, inter-cluster dispersion and intra-cluster dispersion according to the adjusted new clusters; obtaining a distance criterion function according to the new cluster center, the inter-cluster dispersion and the intra-cluster dispersion, selecting the minimum value of the distance criterion function as the number of cluster divisions, and allocating the vector data sets to the corresponding clusters.

[0011] Optionally, after constructing the script category library, the method further includes: when a new category exists, updating the new category to the script category library; retrieving enhanced knowledge related to the new category from an external knowledge graph, and updating the enhanced knowledge to the internal knowledge base.

[0012] Optionally, enhanced knowledge related to the new category is retrieved from an external knowledge graph, including: obtaining the target entity corresponding to the external knowledge graph and the new category; wherein the external knowledge graph includes an entity layer and an instance layer, the entity layer includes a number of entities with associated relationships, and the instance layer includes instances corresponding to each entity; according to the external knowledge graph, the first-order neighbor entity of the target entity is obtained; wherein the first-order neighbor entity is an entity directly associated with the target entity; and the instance corresponding to the target entity and the instance corresponding to the first-order neighbor are used as enhanced knowledge related to the new category.

[0013] Optionally, the method also includes: calculating the vector distance between the target entity and each entity in the external knowledge graph; calculating the similarity between the target entity and each entity in the external knowledge graph based on the vector distance; obtaining an extended entity of the target entity based on the similarity between the target entity and each entity in the external knowledge graph; obtaining the first-order neighbor entity of the extended entity based on the external knowledge graph; and using the instance corresponding to the extended entity and the instance corresponding to the first-order neighbor entity of the extended entity as enhanced knowledge related to the new category.

[0014] According to one aspect of an embodiment of the present application, a script classification device is provided, the device comprising: a database construction module, for constructing a script knowledge base, a script intent library and a script category library based on internal documents and script data sets; a candidate class acquisition module, for acquiring relevant documents from the script knowledge base, relevant intents from the script intent library and relevant candidate classes from the category library according to the script to be classified; a thought chain construction module, for constructing a thought chain based on the relevant documents, the relevant intents and the candidate classes; and a category analysis module, for inputting the thought chain into a target large language model for analysis to obtain a target category corresponding to the script to be classified.

[0015] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the script classification method in the above technical solution is implemented.

[0016] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions so that the electronic device implements the script classification method in the above technical solution.

[0017] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the script classification method in the above technical solution is implemented.

[0018] The technical solution provided by the present application includes at least the following beneficial effects: the present application constructs a thinking chain by respectively extracting relevant intents, relevant documents and candidate classes that match the script to be classified from the script intent library, the script knowledge base and the script category library, so that the large language model uses the relevant information in the thinking chain to identify the script category; therefore, the present application realizes the intelligent classification of scripts generated by the operation and maintenance within the enterprise. When the script is input, some candidate classes will be given, and the input script will be automatically described. The reasoning ability of the large language model itself and the relevant documents will be used to identify the intent description associated with the input script, thereby improving the efficiency of script classification while improving the accuracy of script classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0020] Figure 1 The figure is a flow chart of a script classification method provided in an embodiment of the present application;

[0021] Figure 2 Shown Figure 1 An exemplary flow chart of step S10 in FIG.

[0022] Figure 3 The figure shows a logical framework diagram of constructing a script intent library and a script knowledge base provided by an embodiment of the present application;

[0023] Figure 4 The figure shows a sample schematic diagram of a thinking chain provided in an embodiment of the present application;

[0024] Figure 5 Shown is a schematic diagram of an output sample of a target large language model provided in an embodiment of the present application;

[0025] Figure 6 Shown is a flow chart of another script classification method provided in an embodiment of the present application;

[0026] Figure 7 The figure is a schematic diagram of a process of a vector clustering method provided in an embodiment of the present application;

[0027] Figure 8 The figure is a schematic diagram of the structure of a script classification device provided in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0030] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0031] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0032] It should also be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0033] In the production and operation and maintenance field of Internet companies, the classification of operation and maintenance scripts is a key link in standardized processing. Operation and maintenance scripts usually contain multiple parts such as scene commands, parameters, function definitions and explanations. The simplest line of command can be a script. Companies need to standardize and classify and review the source code of products and scripts running in information systems. Specifically, front-line operation and maintenance personnel in the industry spend nearly half of their time on production equipment every day manually writing execution scripts consisting of multiple commands to operate production equipment. However, this process involves production operation risks. In the audit of these production operations, the scripts need to be labeled and classified into the operation action categories in the management specifications in order to conduct corresponding audits. Therefore, clarifying the user permissions and risk levels involved in the scripts is crucial to ensuring code security and permission privacy.

[0034] However, in the real production environment, script classification is still mainly based on manual labeling. Skilled business personnel label the scripts and classify them into a series of categories with specific and clear functions. The category label is the operation ID in the management specification and will be updated over time and with business. For example, an operation and maintenance script can be divided into operation ID-45 by auditors according to the management specification. The operation is a tuple composed of multiple subcategories, which can correspond to different contents, such as the first-class operation object application server / container, the second-class object system log, and the operation action query. That is, a category triplet can be expressed as (application server / container, system log, query), or application server / container-system log-query. Then, audit work is carried out based on these labels to determine the applicable position, applicable department, operation level and other information of the operation and maintenance script to ensure the security and compliance of the script, and then the business personnel or audit department submits it to the downstream task.

[0035] However, implementing script classification by manual annotation not only has the problems of high cost and low annotation efficiency, but also the classification annotations obtained by different personnel based on personal experience are subjective and unstable, thus affecting the accuracy of script classification. In order to solve the above problems, the present application provides a script classification method based on a large language model, which specifically includes the following embodiments:

[0036] Figure 1 FIG. 1 is a flow chart of a script classification method provided in an embodiment of the present application; Figure 1 As shown, the method specifically comprises the following steps:

[0037] Step S10: construct a script knowledge base, a script intent library and a script category library based on internal documents and script data sets.

[0038] Specifically, it includes: building a script category library, building a script knowledge base based on internal documents, and building a script knowledge base based on script datasets.

[0039] In this embodiment, internal documents include code writing documents that operation and maintenance personnel may come into contact with, specification documents of different systems, user manuals of common tools, and other resources. Through data engineering, sensitive information is removed, duplicate values ​​are removed, and blank values ​​are processed to obtain cleaned internal documents. A script knowledge base based on document context is constructed with documents as nodes and the relationships between documents as edges. Among them, the script knowledge base can improve the large language model's ability to understand and effectively utilize script knowledge.

[0040] Optionally, in order to increase the semantic connection between code and text, the present application can also use punctuation processing, unit length segmentation, document clustering and other means to generate mixed code and text knowledge fragments, such as code segments containing text comments, code segments containing script intent descriptions, etc., for the script knowledge base based on the document context obtained in the above steps, and store them in the form of markdown (lightweight markup language). This method can obtain a script knowledge base with rich script code knowledge, thereby improving the classification effect.

[0041] Figure 2 Shown Figure 1 An exemplary flow chart of step S10 in FIG. Figure 2 As shown in the figure, building a script intent library based on the script dataset specifically includes the following steps:

[0042] Step S11, obtaining a first statistical feature of the closed-source script data set and a second statistical feature of the open-source script data set, the statistical features including script length, script language, and commonly used commands of the script.

[0043] It should be noted that the script dataset includes closed-source script datasets and open-source script datasets. The closed-source script dataset is real script data written by operation and maintenance personnel within the enterprise, as a few-sample scenario; the open-source script dataset is crawled and screened from the developer code hosting platform Github and the code question and answer online community Stack Overflow&Stack Exchange, as well as code teaching websites such as the novice tutorial Runoob and other websites, which may involve knowledge about script commands, question and answer-explanation data sample pairs, and operation and maintenance-related operation knowledge. This part of knowledge is also converted into markdown form to store the context; then the open-source dataset is used to screen the command data of the python scripts and shell scripts contained therein, remove irrelevant field features, and use some summary, explanation and other fields as pseudo-labels to construct this application as an open-source script dataset for a multi-sample scenario.

[0044] In this embodiment, the distribution of surface information such as the length of the internal real script, the language system to which the script belongs, and the commonly used commands of the script (referring to the basic commands that often appear in the internal script but are not in the Shell and Python language systems) is calculated by statistical methods; specifically, the script length is calculated by the file size, number of code lines, and code character length of the real script through the editing tool; the script language (Python, Shell) is determined by the script file header, file suffix, and script content keywords; the commonly used commands of the operation and maintenance script are determined by calculating word frequency, keyword extraction, and rule-based practices. Combining the above practices, this embodiment can obtain the first statistical feature of closed-source script data in the internal environment and the second statistical feature of open-source script data in the external environment; wherein the first statistical feature and the second statistical feature respectively include the script length, script language, and commonly used commands of the script.

[0045] Step S12: According to the distribution difference between the first statistical feature and the second statistical feature, script data related to the closed-source script dataset is obtained from the open-source script dataset to obtain an external script dataset.

[0046] It should be noted that although a large amount of open source script data is obtained in the above steps, it is not necessarily related to the internal script data. In order to reduce data interference and processing efficiency, this embodiment measures the distribution difference of script data by calculating the KL divergence between the first statistical feature and the second statistical feature, and extracts script data with smaller distribution difference from the closed-source script data set from the open source script data set as the external script data set.

[0047] Step S13: Obtain a matching intention recognition model based on the first statistical feature.

[0048] It should be noted that in order to improve the accuracy and efficiency of intent recognition, this embodiment matches the script length in the first statistical feature with the context length supported by the language model, evaluates whether the script language in the first statistical feature and the language model pre-training have relevant language data, and selects a suitable intent recognition model.

[0049] Step S14: perform intent recognition on the closed-source script dataset and the external script dataset respectively according to the intent recognition model to obtain internal commonly used command intent pairs and external common command intent pairs.

[0050] Specifically, the internal common command-intent pair includes the common command and its intent, and the external common command-intent pair includes the common command and its intent.

[0051] It should be noted that at this stage, it is necessary to extract the common intent of the open source scripts, that is, the common commands and their intents of different Linux systems. The intent recognition model can be used for reasoning or the API of the commercial large model can be called to extract the common intent. After completing the common intent extraction operation, the corresponding open source script-intent pair is obtained. Due to the large amount of open source data, this embodiment also needs to reduce the dimension and cluster the common intent according to the features such as commands and templates, such as clustering algorithms and topic models in sampling machine learning. After this operation, representative external common command intent pairs are obtained, on which a basic intent knowledge base can be built.

[0052] At the same time, by deploying the intent recognition model in the internal test environment to identify the intent of internal scripts (i.e. closed-source scripts), prompt engineering is used to extract internal commonly used commands with high operation frequency, high degree of repetition, and industry user preferences from massive internal operation and maintenance scripts, thereby obtaining internal commonly used command intention pairs.

[0053] Step S15: construct a script intent library based on internal commonly used command intent pairs and external general command intent pairs.

[0054] In one embodiment, a basic intent library is constructed based on external common command intention pairs, and the basic intent library is improved by internal commonly used command intention pairs, that is: the internal commonly used command intention pairs are matched with the basic intent library, and if there are commonly used commands that are different from the common commands, the commonly used command intention pairs are inserted into the basic intent library, thereby forming a script intent library.

[0055] In another embodiment, there are internal special commands in the internal script, such as common commands with the wx- prefix, and their intents may not be extracted, and only the situation where the intent is empty can be obtained; therefore, when there is a special command with an empty intent in the internal common command intent, the method also includes: preprocessing the special command to obtain the target command; obtaining the target intent and Internet document corresponding to the target command based on the Internet search, and inserting the target intent into the script intent library and the Internet document to update the script knowledge base.

[0056] It should be noted that preprocessing of special commands includes removing the predefined prefix or suffix, and then obtaining the intent corresponding to the preprocessed target command through Internet search, and inserting the intent into the script intent library, thereby improving the script intent library; in addition, the present application also uses Internet search to obtain Internet documents corresponding to the target command in the manner of obtaining open source script data sets, and updates them to the script knowledge base.

[0057] In summary, the logical framework of this application to construct the script intent library and script knowledge base is as follows: Figure 3As shown, after the collected open source script data is parsed for general intent through the intent recognition model, an initial script intent library consisting of external general command intent pairs is obtained. At the same time, the closed-source script data is analyzed for common intent through the intent recognition model to obtain internal common command intent pairs, and then the internal common command intent pairs are matched with all external common command intent pairs in the script intent library. If they can be matched, the internal common command intent pairs are the external common command intent pairs. If they cannot be matched, the internal common command intent pairs are inserted into the script intent library. If there are special commands with empty intents, the corresponding intents are searched on the Internet to complete them in the script intent library. Finally, professionals conduct quality assessment on the intent pairs corresponding to the internal common commands, manually correct the corresponding real intents, and finally improve the script knowledge base and script intent library to prepare knowledge support for the construction of the subsequent retrieval enhancement framework.

[0058] Step S20: Based on the script to be classified, relevant documents are extracted from the script knowledge base, relevant intents are obtained from the script intent library, and relevant candidate classes are obtained from the category library.

[0059] In this embodiment, the relevant documents extracted from the script knowledge base, the relevant intentions obtained from the script intention library, and the relevant candidate classes obtained from the category library based on the script to be classified specifically include: vectorizing the script knowledge base, the script intention library and the script category library to obtain a vector cluster corresponding to each script category; obtaining at least two candidate vector clusters matching the target vector according to the center distance from the target vector corresponding to the script to be classified to each vector cluster; obtaining relevant documents, relevant intentions and relevant candidate classes corresponding to the script to be classified from the at least two candidate vector clusters.

[0060] It should be noted that a fine-tuning module is designed, which will apply an existing lightweight language model (such as BERT) to learn features related to tasks and data, such as key instructions and key comments in the script, which are information with high value for classifying the script; the fine-tuning module projects the relevant intent, relevant documents and candidate class information of scripts of different categories into the vector space using a lightweight language model trained by contrastive learning, and then uses the k-means method to generate clusters corresponding to each category in the vector space. When classifying the script, the present invention first uses a lightweight language model trained by contrastive learning to project the script to be classified into the vector space, and then refers to its distance from the cluster center corresponding to each category in the vector space, selects some clusters with the closest distance, and then uses the relevant intent, relevant documents and candidate class information of the script corresponding to these clusters as the relevant documents, relevant intent and relevant candidate classes corresponding to the script to be classified.

[0061] Step S30: construct a thought chain according to relevant documents, relevant intentions and candidate classes.

[0062] This embodiment will use the relevant intentions extracted from the script intention library, combined with the relevant documents extracted from the enhanced script knowledge base, and the candidate classes screened from the script category library to construct a thinking chain sample such as Figure 4 As shown in the figure, the thought chain includes task description, input script (i.e., script to be classified), relevant intent, relevant documents, candidate classes, and reasoning steps. The thought chain is used as the input of the large language model to generate the classification ideas and reasoning process of the large language model, so that the large language model can achieve better performance in subsequent classification of similar data with reliable explanations.

[0063] In addition, when constructing a thought chain, the order in which script-related intentions, related documents, and candidate class information belonging to different clusters are arranged in the instruction will be determined by the distance between the input script and the corresponding cluster of this information. The closer to a cluster, the higher the ranking of the relevant intentions, related documents, and candidate class information corresponding to the cluster in the instruction; for example: <Related command 1, intention Figure 1 >, “Number 1, document context 1”, and the clusters corresponding to category 1 are closest to the vector space distance of the input script.

[0064] Step S40: input the thought chain into the target large language model for analysis to obtain the target category corresponding to the script to be classified.

[0065] The invention will combine the chain of thought (CoT) technology to enable the target large language model to read the information in the constructed chain of thought instructions, and think and judge based on this information, and output the chain of thought of the judgment process, so that the large model can better use a variety of related information to perform script classification tasks; for data input, the large model constructs the chain of thought (CoT) by itself, and completes the specified task according to the chain of thought. Figure 5 shown.

[0066] The technical solution provided in this embodiment has at least the following beneficial effects:

[0067] The present application constructs a thinking chain by respectively extracting relevant intents, relevant documents and candidate classes that match the script to be classified from the script intent library, the script knowledge base and the script category library, so that the large language model uses the relevant information in the thinking chain to identify the script category; therefore, the present application realizes the intelligent classification of scripts generated by the operation and maintenance within the enterprise. When the script is input, some candidate classes will be given, and the input script will be automatically described. The reasoning ability of the large language model itself and the relevant documents will be used to identify the intent description associated with the input script, thereby improving the efficiency of script classification while improving the accuracy of script classification.

[0068] In another embodiment of the present application, after constructing the script category library, the method further includes: when a new category exists, updating the new category to the script category library; retrieving enhanced knowledge related to the new category from the external knowledge graph, and updating the enhanced knowledge to the internal knowledge base.

[0069] It should be noted that after the script knowledge base and script intent library are constructed, this solution will use retrieval enhancement generation technology (RAG) to assist in the prediction of the newly added categories. Since when a new category is added, the new category data is often less than the previously existing categories, the prediction effect of the newly added category will be poor. Therefore, the main purpose of this embodiment is to efficiently utilize and update the script knowledge base and script intent library constructed in the previous steps, thereby enhancing the rationality of the generation of the thinking chain; Figure 6 As shown in the figure, first, the category library will be dynamically updated when the category is updated; at the same time, relevant knowledge will be retrieved in the external knowledge graph by calculating the term frequency-inverse text frequency index (TF-IDF) and the code vector dot product similarity, so as to provide a knowledge basis for the subsequent enhancement of the script knowledge base; then, the knowledge in the external knowledge graph will be used to supplement the relevant information of the new category, and the additional knowledge in the external knowledge graph will be added to the script knowledge base, so as to enhance the knowledge of the script knowledge base and make it have richer information related to the new category; finally, the relevant intents extracted from the script intent library will be used, combined with the relevant documents extracted from the enhanced script knowledge base and the candidate classes in the script category library to construct the instruction input, set the few-sample and zero-sample scenarios, and let the large language model construct the chain of thought (CoT) by itself, generate the classification ideas and reasoning process of the large language model, so that the large language model can obtain better performance in the subsequent classification of similar data, with reliable explanations.

[0070] As the input length shows, the thought chain itself cannot contain all relevant information in some cases; therefore, the present application can also inject the extracted relevant intentions, relevant documents and candidate classes into the fine-tuning module in the form of parameter fine-tuning and comparative learning. The fine-tuning module will project this information into the vector space to form clusters. Later, when classifying the script, the fine-tuning module will first encode the script into a vector in the vector space, and then find the clusters close to it in the vector space, so as to extract the relevant intentions, relevant documents and candidate classes in these clusters, and generate their corresponding confidence scores. Instructions are constructed through this information to assist the large model in category prediction.

[0071] like Figure 6As shown, this application will design a training mechanism that only uses the large model as an information source and predictor during training, thereby bypassing the fine-tuning of the large language model and only training the fine-tuning module. After each category update, this application will use a contrastive learning method to project the obtained relevant intent and relevant document information into the vector space. Specifically, the distance between the relevant intent and relevant document information of scripts of different categories in the vector space will be maximized during training. At each update, this application will maximize the distance between the new category and other different categories and irrelevant old categories in the vector space. Using this differentiated contrastive learning method, the new category information is incrementally trained into the fine-tuning module, so that the fine-tuning module receives new category information while retaining the old category information. After obtaining relevant data from multiple sources, it can efficiently integrate the knowledge of relevant data from multiple sources into the fine-tuning module without updating the parameters of the large model itself, saving time for fine-tuning a large number of parameters, and avoiding the catastrophic forgetting problem of the large model.

[0072] In one embodiment of the present application, Figure 7 As shown, the script knowledge base, script intent library and script category library are vectorized to obtain vector clusters corresponding to each script category, which specifically includes the following steps:

[0073] Step S21, extracting vectors from the data in the script knowledge base, the script intent base, and the script category base to obtain vector data sets corresponding to different categories;

[0074] Step S22, selecting K initial cluster centers according to the mean and standard deviation of the vector data set;

[0075] Step S23, calculating the distance between each vector data and each cluster, and dividing each vector data into the cluster with the minimum distance;

[0076] Step S24, calculating new cluster centers, inter-cluster dispersion and intra-cluster dispersion according to the adjusted new clusters;

[0077] Step S25, obtaining a distance criterion function according to the new cluster center, inter-cluster dispersion and intra-cluster dispersion, selecting a K value that minimizes the distance criterion function as the number of clusters, and assigning the vector data set to the corresponding cluster.

[0078] It should be noted that, assuming that the vector data set includes N three-dimensional vector data to be clustered, the specific steps for vector clustering of adjacent data sets are: (a) Calculate the mean μ and standard deviation σ of all vector data, that is, the data is mainly distributed between (μ-σ, μ+σ), and select K initial cluster centers in this interval; (b) According to the Euclidean distance formula, calculate the distance from each vector data to each cluster, and divide each data into the cluster with the minimum distance; (c) Calculate the new cluster center for the adjusted new cluster. If there is no change in the cluster centers of two adjacent clusters, it means that the data object adjustment is completed; (d) Since the K value is given in advance and may not be the optimal solution, based on the principle of maximum inter-cluster discreteness and minimum intra-cluster discreteness, use the criterion function to further optimize the K value. Take the determined cluster center as the initial cluster center, calculate the distance between each data object and the initial cluster center, and calculate the distance criterion function. The K value that minimizes the criterion function value is used as the number of final cluster divisions. (e) Reassign the spatial objects to the corresponding clusters according to the Euclidean distance formula and update the centers of each cluster until the clustering result remains unchanged.

[0079] In one embodiment of the present application, enhanced knowledge related to a new category is retrieved from an external knowledge graph, including: obtaining a target entity e corresponding to the external knowledge graph and the new category; obtaining first-order neighbor entities of the target entity based on the external knowledge graph; wherein the first-order neighbor entities are entities directly associated with the target entity; and using instances corresponding to the target entity and instances corresponding to the first-order neighbors as enhanced knowledge related to the new category.

[0080] It should be noted that the external knowledge graph in this embodiment is constructed using known technologies, wherein the external knowledge graph includes an entity layer and an instance layer, the entity layer includes a number of entities with association relationships, and the instance layer includes instances corresponding to each entity; let the set of all entities in the entity layer be E = (e 1 ,e 2 ,…,e n ); Use the trained entity recognition model to extract the target entity e in the new category retrieval statement, match the target entity e with the entities in the external knowledge graph, that is, find the entity e i ∈E, such that e=e i , that is, find the entity e that is the same as the target entity in the external knowledge graph i , and then obtain the entity e according to the association relationship of the entity layer i Directly connected first-order neighbor entities; entity e i The corresponding instances and the instances corresponding to the first-order neighboring entities serve as enhanced knowledge related to the new category.

[0081] Furthermore, in order to further expand the search scope and obtain richer enhanced knowledge, the present application also includes calculating the vector distance between the target entity and each entity in the external knowledge graph; calculating the similarity between the target entity and each entity in the external knowledge graph based on the vector distance; obtaining the extended entity of the target entity based on the similarity between the target entity and each entity in the external knowledge graph; obtaining the first-order neighbor entity of the extended entity based on the external knowledge graph; and using the instance corresponding to the extended entity and the instance corresponding to the first-order neighbor entity of the extended entity as enhanced knowledge related to the new category.

[0082] It should be noted that the Manhattan distance is used to calculate the distance between the target entity e and the j-th entity e in the external knowledge graph. j The vector distance d between j , the specific calculation formula is:

[0083]

[0084] Then use the vector distance d j Calculate the target entity e to the j-th entity e in E j The calculation formula of similarity is:

[0085]

[0086] Calculate the target entity e and entity e j After calculating the similarity, we can find several extended entities with the greatest similarity to the target entity e. The extended entity has a strong semantic similarity with the target entity, thereby completing the associative expansion of the new category retrieval sentence entity.

[0087] Figure 8 FIG. 1 is a schematic diagram of the structure of a script classification device provided in an embodiment of the present application; Figure 8 As shown, the device comprises:

[0088] A database construction module 810 is used to construct a script knowledge base, a script intent base and a script category base according to internal documents and script data sets;

[0089] A candidate class acquisition module 820 is used to acquire relevant documents from a script knowledge base, acquire relevant intents from a script intent library, and acquire relevant candidate classes from a category library according to the script to be classified;

[0090] A thought chain building module 830 is used to build a thought chain according to relevant documents, relevant intentions and candidate classes;

[0091] The category analysis module 840 is used to input the thought chain into the target large language model for analysis to obtain the target category corresponding to the script to be classified.

[0092] In one embodiment, the database construction module 810 includes:

[0093] A statistical feature acquisition unit, used to acquire a first statistical feature of a closed-source script data set and a second statistical feature of an open-source script data set, wherein the statistical features include script length, script language, and common commands of the script;

[0094] An external script data set acquisition unit, configured to acquire script data related to the closed-source script data set from the open-source script data set according to a distribution difference between the first statistical feature and the second statistical feature, to obtain an external script data set;

[0095] An intention recognition model acquisition unit, configured to acquire a matching intention recognition model according to the first statistical feature;

[0096] A command-intent pair acquisition unit is used to perform intent recognition on a closed-source script dataset and an external script dataset according to an intent recognition model, and obtain internal commonly used command-intent pairs and external common command-intent pairs;

[0097] The intention library building unit is used to build a script intention library based on internal common command intention pairs and external common command intention pairs.

[0098] In one embodiment, the database construction module 810 further includes:

[0099] A preprocessing unit, used for preprocessing special commands to obtain target commands;

[0100] The search unit is used to obtain the target intent and Internet documents corresponding to the target command based on the Internet search, and insert the target intent into the script intent library and update the Internet documents to the internal knowledge base.

[0101] In one embodiment, the candidate class acquisition module 820 includes:

[0102] A vector clustering unit is used to vectorize the script knowledge base, the script intent base and the script category base to obtain vector clusters corresponding to each script category;

[0103] A center distance matching unit, used to obtain at least two candidate vector clusters matching the target vector according to the center distance from the target vector corresponding to the script to be classified to each vector cluster;

[0104] The candidate class acquisition unit is used to acquire relevant documents, relevant intents and relevant candidate classes corresponding to the script to be classified from at least two candidate vector clusters.

[0105] In one embodiment, the vector clustering unit is also used to extract vectors from data in a script knowledge base, a script intent library, and a script category library to obtain vector data sets corresponding to different categories; select K initial cluster centers based on the mean and standard deviation of the vector data sets; calculate the distance from each vector data to each cluster, and divide each vector data into a cluster with a minimum distance; calculate new cluster centers, inter-cluster dispersion, and intra-cluster dispersion based on the adjusted new clusters; obtain a distance criterion function based on the new cluster centers, inter-cluster dispersion, and intra-cluster dispersion, select the minimum value of the distance criterion function as the number of clusters to be divided, and distribute the vector data sets to the corresponding clusters.

[0106] In one embodiment, the apparatus further comprises:

[0107] A new category update module, used for updating the new category to the script category library when there is a new category;

[0108] The knowledge graph retrieval module is used to retrieve enhanced knowledge related to new categories from the external knowledge graph and update the enhanced knowledge to the internal knowledge base.

[0109] In one embodiment, the knowledge graph retrieval module includes:

[0110] A target entity acquisition unit is used to acquire the target entity corresponding to the external knowledge graph and the new category; wherein the external knowledge graph includes an entity layer and an instance layer, the entity layer includes a number of entities with association relationships, and the instance layer includes instances corresponding to each entity;

[0111] A first neighbor entity acquisition unit is used to acquire first-order neighbor entities of a target entity according to an external knowledge graph; wherein the first-order neighbor entities are entities directly associated with the target entity;

[0112] The first instance acquisition unit is used to use the instance corresponding to the target entity and the instance corresponding to the first-order neighbor as enhanced knowledge related to the new category.

[0113] In one embodiment, the knowledge graph retrieval module further includes:

[0114] A vector distance calculation unit is used to calculate the vector distance between the target entity and each entity in the external knowledge graph;

[0115] A similarity calculation unit is used to calculate the similarity between the target entity and each entity in the external knowledge graph based on the vector distance;

[0116] An extended entity acquisition unit, used to acquire an extended entity of a target entity according to the similarity between the target entity and each entity in an external knowledge graph;

[0117] A second neighbor entity acquisition unit is used to acquire the first-order neighbor entities of the extended entity according to the external knowledge graph;

[0118] The second instance acquisition unit is used to use the instance corresponding to the extended entity and the instance corresponding to the first-order neighboring entity of the extended entity as enhanced knowledge related to the new category.

[0119] Fig. 9 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0120] It should be noted that Fig. 9 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0121] like Fig. 9 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003, such as executing the method described in the above embodiment. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002 and the RAM 1003 are connected to each other through the bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.

[0122] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0123] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1009, and / or installed from a removable medium 1011. When the computer program is executed by a central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.

[0124] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0125] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0126] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0127] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A script classification method, characterized in that: The method comprises: Build a script knowledge base, script intent base, and script category base based on internal documents and script datasets; According to the script to be classified, obtaining relevant documents from the script knowledge base, obtaining relevant intents from the script intent library, and obtaining relevant candidate classes from the category library; Construct a thought chain according to the relevant documents, the relevant intentions and the candidate classes; The thought chain is input into the target large language model for analysis to obtain the target category corresponding to the script to be classified.

2. The method according to claim 1, characterized in that When the script data set includes a closed-source script data set and an open-source script data set, constructing a script intent library according to the script data set includes: Obtaining a first statistical feature of the closed-source script data set and a second statistical feature of the open-source script data set, wherein the statistical features include script length, script language, and common commands of the script; According to the distribution difference between the first statistical feature and the second statistical feature, obtaining script data related to the closed-source script dataset from the open-source script dataset to obtain an external script dataset; According to the first statistical feature, obtaining a matching intention recognition model; According to the intention recognition model, the closed-source script dataset and the external script dataset are respectively subjected to intention recognition to obtain internal common command intention pairs and external common command intention pairs; The script intent library is constructed according to the internal commonly used command intent pairs and the external general command intent pairs.

3. The method according to claim 2, characterized in that When there is a special command with an empty intent in the internal common command intent pair, the method further includes: Preprocessing the special command to obtain a target command; The target intent and Internet document corresponding to the target command are obtained through Internet search, and the target intent is inserted into the script intent library and the Internet document is updated to the internal knowledge base.

4. The method according to claim 1, characterized in that: According to the script to be classified, obtaining relevant documents from the script knowledge base, obtaining relevant intents from the script intent library, and obtaining relevant candidate classes from the category library include: Vectorizing the script knowledge base, the script intent library, and the script category library to obtain vector clusters corresponding to each script category; According to the center distance between the target vector corresponding to the script to be classified and each vector cluster, obtaining at least two candidate vector clusters matching the target vector; Relevant documents, relevant intents and relevant candidate classes corresponding to the script to be classified are obtained from the at least two candidate vector clusters.

5. The method according to claim 4, characterized in that The script knowledge base, the script intent library and the script category library are vectorized to obtain vector clusters corresponding to each script category, including: Performing vector extraction on the data in the script knowledge base, the script intent library and the script category library to obtain vector data sets corresponding to different categories; Selecting K initial clustering centers according to the mean and standard deviation of the vector data set; Calculate the distance between each vector data and each cluster, and divide each vector data into the cluster with the minimum distance; Calculate new cluster centers, inter-cluster dispersion, and intra-cluster dispersion based on the adjusted new clusters; A distance criterion function is obtained according to the new cluster center, the inter-cluster dispersion and the intra-cluster dispersion, a minimum value of the distance criterion function is selected as the number of cluster divisions, and the vector data sets are allocated to corresponding clusters.

6. The method according to any one of claims 1 to 5, characterized in that: After building the script category library, the method further includes: When a new category exists, updating the new category to the script category library; Retrieve enhanced knowledge related to the new category from the external knowledge graph, and update the enhanced knowledge to the internal knowledge base.

7. The method according to claim 6, characterized in that Retrieve enhanced knowledge related to the new category from the external knowledge graph, including: Obtaining an external knowledge graph and a target entity corresponding to the new category; wherein the external knowledge graph includes an entity layer and an instance layer, the entity layer includes a number of entities with association relationships, and the instance layer includes instances corresponding to each entity; According to the external knowledge graph, the first-order neighbor entity of the target entity is obtained; wherein the first-order neighbor entity is an entity directly associated with the target entity; The instance corresponding to the target entity and the instance corresponding to the first-order neighbor are used as enhanced knowledge related to the new category.

8. The method according to claim 7, characterized in that The method further comprises: Calculate the vector distance between the target entity and each entity in the external knowledge graph; Calculate the similarity between the target entity and each entity in the external knowledge graph based on the vector distance; According to the similarity between the target entity and each entity in the external knowledge graph, obtaining the extended entity of the target entity; According to the external knowledge graph, obtain the first-order neighbor entity of the extended entity; The instance corresponding to the extended entity and the instance corresponding to the first-order neighboring entity of the extended entity are used as enhanced knowledge related to the new category.

9. A script classification device, characterized in that: The device comprises: A database construction module is used to construct a script knowledge base, a script intent base and a script category base based on internal documents and script datasets; A candidate class acquisition module is used to acquire relevant documents from the script knowledge base, acquire relevant intents from the script intent library, and acquire relevant candidate classes from the category library according to the script to be classified; A thought chain building module, used for building a thought chain according to the relevant documents, the relevant intents and the candidate classes; The category analysis module is used to input the thought chain into the target large language model for analysis to obtain the target category corresponding to the script to be classified.

10. An electronic device, characterized in that: include: processor; as well as A memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to enable the electronic device to implement the script classification method as described in any one of claims 1 to 8.

Images