Method and device for building multi-dimensional large language model capability framework

Abstract

The invention provides a method and a device for building a multi-dimensional large language model capability framework, and relates to the technical field of large language models. The method comprises the following steps: defining the cognitive dimension ability of a large language model based on the cognitive ability of a CHC theoretical model; defining the domain dimension capability of the large language model based on a FLASK domain classification system; defining the task dimension capability of the large language model; according to the cognitive dimension capability, the field dimension capability and the task dimension capability of the large language model, constructing a multi-dimensional large language model capability framework; a capability labeling training set is obtained, a GPT-4o model is adopted to label the training set, and a labeled data set is obtained; according to the labeled data set, training a capability framework to obtain a trained multi-dimensional capability labeling model; obtaining fine tuning data of the to-be-screened large language model; and inputting the fine tuning data into the trained multi-dimensional capability labeling model to obtain high-quality fine tuning data. According to the invention, high-quality fine-tuning data can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a method and device for building a multi-dimensional large language model capability framework. Background Art

[0002] In recent years, the capabilities of large language models have made significant progress and breakthroughs. In particular, the introduction of reinforcement learning and the application of thought chain reasoning have further enhanced their reasoning capabilities. At present, large language models are widely used in various industries, such as writing, conversation, and programming, to provide convenience for life. As large language models become more and more complex, it is particularly important to accurately evaluate their underlying capabilities so that researchers can perceive the current capabilities of large language models and further expand the capabilities of large language models. At present, there are a variety of general benchmarks that are widely used to evaluate these capabilities. However, many benchmarks only focus on certain aspects of model capabilities, such as programming, common sense reasoning, or performance on specific tasks. For example, Massive Multitask Language Understanding (MMLU), a large-scale multi-task language understanding benchmark, evaluates the knowledge mastery of large language models in academic fields, but ignores dimensions such as code generation. In addition, the capability dimensions of existing benchmarks are usually task-oriented, and there is no comprehensive framework to systematically classify and summarize the capabilities of large language models, which makes the current industry lack a comprehensive and systematic understanding of the capabilities of large language models.

[0003] In recent years, there have been many studies at home and abroad exploring the capabilities of large language models. Some studies focus on model evaluation, defining evaluation criteria from three aspects: domain, capability, and difficulty, measuring the model's response generation, and evaluating the model's capabilities; some studies do not predefine capabilities, but rely on the model itself to label the data with capability labels; some studies start from a cognitive perspective, dividing model capabilities into four aspects: language knowledge, formal knowledge, world modeling, and social modeling, and constructing test sets from the above four aspects to evaluate model capabilities; some studies explore the relationship between a single model capability and cross-capability. The above studies either focus on measuring model capabilities in a single or limited dimension, such as starting from task definition alone, or lack a detailed definition and decoupling of capabilities, and the definition of the capabilities of large language models is not systematic and comprehensive enough. In addition, most studies apply the definition of capabilities to the capability evaluation of large language models, and lack other explorations of the application of capability frameworks. Summary of the invention

[0004] In order to solve the technical problems that the existing technology has limited definition of the capabilities of large language models, lacks a multi-dimensional perspective, and lacks exploration of the application of the capability framework, resulting in accurate low-level prediction in data screening, the embodiment of the present invention provides a method and device for building a multi-dimensional large language model capability framework. The technical solution is as follows:

[0005] On the one hand, a method for building a multi-dimensional large language model capability framework is provided, the method is implemented by a device for building a multi-dimensional large language model capability framework, and the method includes:

[0006] S1. Based on the multimodal human cognitive ability of the CHC theoretical model, the language ability is screened out; based on the language ability, the domain-related abilities and the non-core abilities of the large language model are removed to preliminarily obtain the cognitive dimension ability of the large language model; based on the preliminarily obtained cognitive dimension ability of the large language model, the inductive ability is refined into pattern recognition ability, concept abstraction ability and hypothesis generation ability, and finally the cognitive dimension ability of the large prediction model is defined;

[0007] S2. Based on the FLASK domain classification system, define the domain dimension capabilities of the large language model; define the task dimension capabilities of the large language model;

[0008] S3. Construct a multi-dimensional large language model capability framework based on the cognitive dimension capability of the large language model, the domain dimension capability of the large language model, and the task dimension capability of the large language model;

[0009] S4. Obtain a capability labeling training set, use the GPT-4o model to perform fine-grained capability labeling on each instruction in the capability labeling training set, and obtain a labeled data set;

[0010] S5. According to the annotated data set, the Qwen2.5-7B-Base model is used to train the multi-dimensional large language model capability framework to obtain a trained multi-dimensional capability annotation model; the fine-tuning data of the large language model to be screened is obtained; the fine-tuning data of the large language model to be screened is input into the trained multi-dimensional capability annotation model to obtain high-quality fine-tuning data.

[0011] Optionally, the multi-dimensional large language model capability framework is expressed by the following formula (1):

[0012] (1)

[0013] in, represents the multi-dimensional large language model capability framework; c represents cognitive ability; d represents domain; t represents task; represents the cognitive dimension; Represents the domain dimension; Represents the task dimension.

[0014] Optionally, the formula of the cognitive dimension is expressed by the following formula (2):

[0015] (2)

[0016] in, Indicates specific cognitive abilities; represents the total number of cognitive dimensions;

[0017] The formula of the domain dimension is expressed by the following formula (3):

[0018] (3)

[0019] in, Indicates a segmented area; Indicates the total number of domain dimensions;

[0020] The formula of the task dimension is expressed by the following formula (4):

[0021] (4)

[0022] in, Represents the defined task; Indicates the total number of task dimensions.

[0023] Optionally, the S4 uses a GPT-4o model to perform fine-grained capability label annotation on each instruction in the capability annotation training set to obtain an annotated data set, including:

[0024] According to the ability labeling training set, two cognitive dimension ability labels are labeled for each data through the GPT-4o model, one domain dimension ability label is labeled for each data through the GPT-4o model, and one task dimension ability label is labeled for each data through the GPT-4o model; by labeling the cognitive dimension ability, domain dimension ability and task dimension ability of each data, the labeled data set is obtained.

[0025] Optionally, the trained multi-dimensional capability labeling model includes: a trained cognitive dimension capability labeling model, a trained domain dimension capability labeling model and a trained task dimension capability labeling model.

[0026] Optionally, the obtaining of fine-tuning data of the large language model to be screened; inputting the fine-tuning data of the large language model to be screened into a trained multi-dimensional capability labeling model to obtain high-quality fine-tuning data includes:

[0027] Input the fine-tuning data of the large language model to be screened into the trained multi-dimensional ability annotation model, and obtain the annotated data pool by annotating the cognitive dimension ability, domain dimension ability, and task dimension ability;

[0028] Based on the annotated data pool, define all the composite capabilities in the annotated data pool; where the composite capabilities are the combination of cognitive dimension capabilities, domain dimension capabilities, and task dimension capabilities;

[0029] Fine-tuning data is screened based on all the composite capabilities in the annotated data pool to obtain high-quality fine-tuning data.

[0030] Optionally, the process of defining all composite capabilities in the labeled data pool is represented by the following formula (5):

[0031] (5)

[0032] in, Represents all composite capabilities in the annotated data pool; Represents the annotated data pool; A composite capability set representing a labeled data pool.

[0033] On the other hand, a device for building a multi-dimensional large language model capability framework is provided, and the device is applied to a method for building a multi-dimensional large language model capability framework. The device includes:

[0034] The first definition unit is used to screen out language ability based on the multimodal human cognitive ability of the CHC theoretical model; based on language ability, remove domain-related abilities and non-core abilities of the large language model to preliminarily obtain the cognitive dimension ability of the large language model; based on the preliminarily obtained cognitive dimension ability of the large language model, refine the inductive ability into pattern recognition ability, concept abstraction ability and hypothesis generation ability, and finally define the cognitive dimension ability of the large prediction model;

[0035] The second definition unit is used to define the domain dimension capabilities of the large language model based on the FLASK domain classification system; define the task dimension capabilities of the large language model;

[0036] A construction unit, used to construct a multi-dimensional large language model capability framework according to the cognitive dimension capability of the large language model, the domain dimension capability of the large language model, and the task dimension capability of the large language model;

[0037] The labeling unit is used to obtain the capability labeling training set, and use the GPT-4o model to label each instruction in the capability labeling training set with a fine-grained capability label to obtain the labeled data set;

[0038] The screening unit is used to train the multi-dimensional large language model capability framework using the Qwen2.5-7B-Base model according to the labeled data set to obtain a trained multi-dimensional capability annotation model; obtain the fine-tuning data of the large language model to be screened; and input the fine-tuning data of the large language model to be screened into the trained multi-dimensional capability annotation model to obtain high-quality fine-tuning data.

[0039] Optionally, the multi-dimensional large language model capability framework is expressed by the following formula (1):

[0040] (1)

[0041] in, represents the multi-dimensional large language model capability framework; c represents cognitive ability; d represents domain; t represents task; represents the cognitive dimension; Represents the domain dimension; Represents the task dimension.

[0042] Optionally, the formula of the cognitive dimension is expressed by the following formula (2):

[0043] (2)

[0044] in, Indicates specific cognitive abilities; represents the total number of cognitive dimensions;

[0045] The formula of the domain dimension is expressed by the following formula (3):

[0046] (3)

[0047] in, Indicates a segmented area; Indicates the total number of domain dimensions;

[0048] The formula of the task dimension is expressed by the following formula (4):

[0049] (4)

[0050] in, Represents the defined task; Indicates the total number of task dimensions.

[0051] Optionally, the marking unit is used to:

[0052] According to the ability labeling training set, two cognitive dimension ability labels are labeled for each data through the GPT-4o model, one domain dimension ability label is labeled for each data through the GPT-4o model, and one task dimension ability label is labeled for each data through the GPT-4o model; by labeling the cognitive dimension ability, domain dimension ability and task dimension ability of each data, the labeled data set is obtained.

[0053] Optionally, the trained multi-dimensional capability labeling model includes: a trained cognitive dimension capability labeling model, a trained domain dimension capability labeling model and a trained task dimension capability labeling model.

[0054] Optionally, the screening unit is used to:

[0055] Input the fine-tuning data of the large language model to be screened into the trained multi-dimensional ability annotation model, and obtain the annotated data pool by annotating the cognitive dimension ability, domain dimension ability, and task dimension ability;

[0056] Based on the annotated data pool, define all the composite capabilities in the annotated data pool; where the composite capabilities are the combination of cognitive dimension capabilities, domain dimension capabilities, and task dimension capabilities;

[0057] Fine-tuning data is screened based on all the composite capabilities in the annotated data pool to obtain high-quality fine-tuning data.

[0058] Optionally, the process of defining all composite capabilities in the labeled data pool is represented by the following formula (5):

[0059] (5)

[0060] in, Represents all composite capabilities in the annotated data pool; Represents the annotated data pool; A composite capability set representing a labeled data pool.

[0061] On the other hand, a device for building a multi-dimensional large language model capability framework is provided, and the device for building a multi-dimensional large language model capability framework includes: a processor; a memory, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, any one of the methods for building a multi-dimensional large language model capability framework is implemented.

[0062] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the methods for building the above-mentioned multi-dimensional large language model capability framework.

[0063] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0064] Based on the multimodal human cognitive ability of the CHC theoretical model, the language ability is screened out; based on the language ability, the domain-related abilities and non-core abilities of the large language model are removed to preliminarily obtain the cognitive dimension ability of the large language model; based on the preliminarily obtained cognitive dimension ability of the large language model, the inductive ability is refined into pattern recognition ability, concept abstraction ability and hypothesis generation ability, and finally the cognitive dimension ability of the large prediction model is defined; based on the FLASK domain classification system, the domain dimension ability of the large language model is defined; the task dimension ability of the large language model is defined; according to the cognitive dimension ability of the large language model and the domain dimension ability of the large language model, the task dimension ability of the large language model is defined. The multi-dimensional large language model capability framework is constructed by combining the task dimension capabilities of the large language model with the capability annotation training set; the capability annotation training set is obtained, and the GPT-4o model is used to perform fine-grained capability label annotation on each instruction in the capability annotation training set to obtain the annotated data set; according to the annotated data set, the Qwen2.5-7B-Base model is used to train the multi-dimensional large language model capability framework to obtain a trained multi-dimensional capability annotation model; the fine-tuning data of the large language model to be screened is obtained; the fine-tuning data of the large language model to be screened is input into the trained multi-dimensional capability annotation model to obtain high-quality fine-tuning data.

[0065] The present invention can be applied to diversity-based general scenario data screening to obtain high-quality fine-tuning data in general scenarios, or can be applied to capability-oriented specific scenario data screening to obtain high-quality fine-tuning data in specific scenarios; the application can improve the accuracy of data screening. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 It is a flow chart of a method for building a multi-dimensional large language model capability framework provided by an embodiment of the present invention;

[0068] Figure 2 It is a schematic diagram of an overall solution of a method for building a multi-dimensional large language model capability framework provided by an embodiment of the present invention;

[0069] Figure 3 It is a schematic diagram of the architecture of cognitive dimension capability, task dimension capability and domain dimension capability of a method for building a multi-dimensional large language model capability framework provided by an embodiment of the present invention;

[0070] Figure 4It is a device block diagram for building a multi-dimensional large language model capability framework provided by an embodiment of the present invention;

[0071] Figure 5 It is a structural diagram of a device for building a multi-dimensional large language model capability framework provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0073] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0074] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0075] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0076] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0077] The embodiment of the present invention provides a method for building a multi-dimensional large language model capability framework, which can be implemented by a device for building a multi-dimensional large language model capability framework, and the device for building a multi-dimensional large language model capability framework can be a terminal or a server. Figure 1 The flowchart of the method for building a multi-dimensional large language model capability framework is shown in the figure. The processing flow of the method may include the following steps:

[0078] S1. Based on the multimodal human cognitive ability of the CHC theoretical model, screen out language ability; based on language ability, remove domain-related abilities and non-core abilities of the large language model, and preliminarily obtain the cognitive dimension ability of the large language model; based on the preliminarily obtained cognitive dimension ability of the large language model, refine the inductive ability into pattern recognition ability, concept abstraction ability and hypothesis generation ability, and finally define the cognitive dimension ability of the large prediction model.

[0079] Among them, the CHC theoretical model covers multiple modal human cognitive abilities, including vision, hearing and language expression. This application screened out language abilities from the human cognitive abilities covered by the CHC theoretical model through manual screening. The memory-related abilities and domain-related abilities were removed by manual removal. The number of CHC-based abilities was reduced from 82 to 14 through screening.

[0080] Among them, in order to connect with the large language model, the inductive ability is refined, and the total number of cognitive dimension capabilities of the large language model is increased to 16 through refinement.

[0081] S2. Based on the FLASK domain classification system, define the domain dimension capabilities of the large language model; define the task dimension capabilities of the large language model.

[0082] Among them, the FLASK domain classification system is a domain classification system in fine-grained language model evaluation based on alignment skills.

[0083] Among them, the FLASK domain classification system covers 38 specific fields; among them, the business field and the marketing field have similarities, which introduces ambiguity into the capability label model and leads to the discreteness of label distribution. Therefore, this application uses artificial means to adjust the 38 fields in the original classification system to 33 fields.

[0084] Among them, 13 tasks are defined based on the granularity and completeness of the tasks.

[0085] S3. Construct a multi-dimensional large language model capability framework based on the cognitive dimension capabilities of the large language model, the domain dimension capabilities of the large language model, and the task dimension capabilities of the large language model.

[0086] Optionally, the multi-dimensional large language model capability framework is expressed by the following formula (1):

[0087] (1)

[0088] in, represents the multi-dimensional large language model capability framework; c represents cognitive ability; d represents domain; t represents task; represents the cognitive dimension; Represents the domain dimension; Represents the task dimension.

[0089] Optionally, the formula of the cognitive dimension is expressed by the following formula (2):

[0090] (2)

[0091] in, Indicates specific cognitive abilities; represents the total number of cognitive dimensions;

[0092] The formula of the domain dimension is expressed by the following formula (3):

[0093] (3)

[0094] in, Indicates a segmented area; Indicates the total number of domain dimensions;

[0095] The formula of the task dimension is expressed by the following formula (4):

[0096] (4)

[0097] in, Represents the defined task; Indicates the total number of task dimensions.

[0098] S4. Obtain the capability labeling training set, use the GPT-4o model to perform fine-grained capability labeling on each instruction in the capability labeling training set, and obtain the labeled data set.

[0099] Optionally, S4 uses the GPT-4o model to annotate each instruction in the capability annotation training set with a fine-grained capability label to obtain an annotated data set, including:

[0100] According to the ability labeling training set, two cognitive dimension ability labels are labeled for each data through the GPT-4o model, one domain dimension ability label is labeled for each data through the GPT-4o model, and one task dimension ability label is labeled for each data through the GPT-4o model; by labeling the cognitive dimension ability, domain dimension ability and task dimension ability of each data, the labeled data set is obtained.

[0101] In order to reduce positional bias, the present invention randomizes the order of abilities in the prompt words of each data when using the GPT-4o model for annotation. When annotating cognitive abilities, cognitive abilities require in-depth instruction understanding, so the present invention uses the GPT-4o model to generate explanations paired with each label.

[0102] S5. According to the annotated data set, the Qwen2.5-7B-Base model is used to train the multi-dimensional large language model capability framework to obtain a trained multi-dimensional capability annotation model; the fine-tuning data of the large language model to be screened is obtained; the fine-tuning data of the large language model to be screened is input into the trained multi-dimensional capability annotation model to obtain high-quality fine-tuning data.

[0103] Optionally, the trained multi-dimensional capability labeling model includes: a trained cognitive dimension capability labeling model, a trained domain dimension capability labeling model, and a trained task dimension capability labeling model.

[0104] Among them, the Qwen2.5-7B-Base model is used to train the cognitive dimension ability labeling model to obtain a trained cognitive dimension ability labeling model; the Qwen2.5-7B-Base model is used to train the domain dimension ability labeling model to obtain a trained domain dimension ability labeling model; the Qwen2.5-7B-Base model is used to train the task dimension ability labeling model to obtain a trained task dimension ability labeling model;

[0105] The FLASK dataset was used as the capability labeling training set, the training batch size was 32, the cosine learning rate was set to 2e-5, and the full fine-tuning method was adopted. A total of 120 steps of fine-tuning were performed, and the test set was used for evaluation every 40 steps. The GPT-4o annotation was used as the standard to evaluate the accuracy of the model's annotation ability, and the model corresponding to the training steps with the highest accuracy was selected as the final model. 10% of the data was randomly divided from the capability labeling training set as the test set.

[0106] Optionally, obtaining fine-tuning data of the large language model to be screened; inputting the fine-tuning data of the large language model to be screened into a trained multi-dimensional capability annotation model to obtain high-quality fine-tuning data, including:

[0107] Input the fine-tuning data of the large language model to be screened into the trained multi-dimensional ability annotation model, and obtain the annotated data pool by annotating the cognitive dimension ability, domain dimension ability, and task dimension ability;

[0108] Based on the annotated data pool, define all the composite capabilities in the annotated data pool; where the composite capabilities are the combination of cognitive dimension capabilities, domain dimension capabilities, and task dimension capabilities;

[0109] Fine-tuning data is screened based on all the composite capabilities in the annotated data pool to obtain high-quality fine-tuning data.

[0110] Optionally, the process of defining all composite capabilities in the annotated data pool is expressed by the following formula (5):

[0111] (5)

[0112] in, Represents all composite capabilities in the annotated data pool; Represents the annotated data pool; A composite capability set representing a labeled data pool.

[0113] In a feasible implementation, a trained multi-dimensional capability annotation model is used to screen data based on a variety of general scenarios; wherein the general scenarios include providing widely applicable capabilities in a variety of tasks such as dialogue, code generation, and content creation, and enhancing the general intelligent performance of the large language model. The data screening process based on a variety of general scenarios includes:

[0114] (1) Obtain a general scenario data set to be screened; input the general scenario data set to be screened into the trained multi-dimensional capability annotation model, and obtain a general scenario annotated data pool by annotating cognitive dimension capabilities, domain dimension capabilities, and task dimension capabilities; based on the general scenario annotated data pool, define all composite capabilities in the general scenario annotated data pool;

[0115] (2) Select a training data set from the annotated data pool of general scenarios; based on the selected training data set, define the composite capability of the training data set, which is expressed by the following formula (6):

[0116] (6)

[0117] in, Represents the composite capability set of the training dataset; Represents the training dataset.

[0118] (3) Setting a threshold and quantifying diversity by the threshold; wherein the threshold is expressed by the following formula (7):

[0119] (7)

[0120] in, Represents the cardinality of the set, that is, the number of elements; The value of reflects the coverage of unique composite capabilities in the training dataset relative to the entire annotated data pool;

[0121] (4) By adjusting the set threshold so that the threshold is close to 1, a screening index is obtained; based on the screening index, the screened data is obtained.

[0122] Among them, when a piece of data in the annotated data pool can increase R, the composite capability of the data is added to the composite capability set of the training data set, and the data is added to the training data set; when R can no longer be increased, the data corresponding to each composite capability is selected from the annotated data pool on average to the training data set until the training data set reaches the expected data volume.

[0123] In a feasible implementation, a trained multi-dimensional capability labeling model is used for data screening based on capability-oriented specific scenarios; the specific scenarios include tasks such as reading comprehension and medical question-answering, so that the performance of the large language model on specific tasks is more accurate and reliable. The data screening process based on capability-oriented specific scenarios includes:

[0124] (1) Obtain a verification data set for a specific scenario to be screened; input the verification data set for the specific scenario to be screened into the trained multi-dimensional capability labeling model, and obtain a labeled verification data set by labeling cognitive dimension capabilities, domain dimension capabilities, and task dimension capabilities;

[0125] (2) Based on the annotated validation dataset, a composite capability set of the annotated validation dataset is constructed, which is expressed by the following formula (8):

[0126] (8)

[0127] in, represents the composite capability set of the annotated validation dataset; Represents the labeled validation dataset.

[0128] (3) Data screening is performed based on the composite capacity of the validation dataset.

[0129] Among them, the goal of the data screening process based on a specific scenario of capability orientation is to select the corresponding data in the labeled data pool for each composite capability in the composite capability set of the labeled verification data set on average. Among them, in actual operation, the composite capabilities in the composite capability set of the labeled verification data set may be relatively limited, and the corresponding data volume of the composite capabilities in the labeled data pool may not reach the expected data volume. Therefore, the present application will further decompose the composite capabilities in the composite capability set of the labeled verification data set. Specifically, the composite capabilities in the composite capability set of the labeled verification data set are decomposed into binary groups to form a binary capability set, and the binary capability set is further decomposed into separate dimensions to form a unigram capability set; when the composite capability set cannot provide enough data, data screening is performed on the binary capability set, and if the expected data volume is still not reached, data screening is performed on the unigram capability set.

[0130] In a feasible implementation, the basic method, the full data method, the random data method and the INSTAG method are used to verify the effectiveness of the multi-dimensional large language ability framework proposed in this application in data screening of diverse general scenarios and data screening of ability-oriented specific scenarios.

[0131] Among them, the basic method is to directly use the basic large language model for testing without any fine-tuning; the all data method is to use all the data in the data pool to fine-tune the basic model; the random data method is to randomly select data from the data pool to fine-tune the basic model; the INSTAG method is an existing large language model capability framework, and the INSTAGGER annotator corresponding to the framework is used to label the data in the data pool.

[0132] Among them, in the data screening of diverse general scenarios, general scenarios include: ARC-C contains multiple-choice questions, which are scientific questions for grades 3 to 9; MMLU contains a series of questions on 57 topics, ranging from elementary to professional levels, mainly to measure the factual knowledge of large language models; BBH contains a variety of challenging tasks, covering more than 200 subtasks, many of which require higher-order and multi-step reasoning; C-EVAL is a Chinese large language model evaluation framework, covering multiple-choice tasks in multiple fields. Table 1 shows the comparison results of data screening in general scenarios.

[0133] Table 1

[0134]

[0135] Among them, the evaluation results in Table 1 show that the multi-dimensional large language model capability framework achieved the best overall performance in the four benchmark tests, achieved the best results on MMLU and C-EVAL, and achieved the second best results on BBH and ARC-C. At the same time, even if only considering a single capability dimension, the method of the present invention is superior to the random data method and the INSTAG method, highlighting the accuracy of the multi-dimensional large language model capability framework in defining capabilities, and proving the effectiveness of the multi-dimensional large language model capability framework in screening general scenario data based on diversity.

[0136] In a feasible implementation, in a specific capability-oriented scenario, a large language model needs data with specific capabilities to meet the requirements of the corresponding task. In a specific scenario, this application selects three relevant test sets, each of which represents a specific capability dimension;

[0137] Among them, in the cognitive dimension, the MedQA dataset was selected for testing. It is a multiple-choice question dataset related to medicine. Experiments were conducted on its subset, which contains 1,273 test data and 1,272 validation set data. The capabilities required by this dataset in the cognitive dimension are mainly hypothesis generation and concept abstraction; among them, in the domain dimension, four history-related tasks were extracted from MMLU, containing 930 test data and 121 validation set data; among them, in the task dimension, SQuAD was selected as the test task of the task dimension. It is a closed-book question answering and extractive question answering task collected from Wikipedia, containing 10.6k test set data. Since it has no validation set, 200 samples were randomly split from the test set as the validation set. For validation sets with a large amount of data, up to 200 samples were selected from the validation set of each task for annotation and data screening. For those validation sets without sufficient samples, the entire validation set was used. Table 2 shows the comparative results of data screening for specific scenarios.

[0138] Table 2

[0139]

[0140] Table 2 shows the evaluation results of the method of the present invention under data screening in specific scenarios. ACC. represents the accuracy rate; EM represents the exact match rate; F1 represents the harmonic mean of precision and recall rate; the method of the present invention outperforms other methods on the three test sets, achieving significant improvement and proving its effectiveness.

[0141] Among them, Figure 2 It is a schematic diagram of the overall solution of a method for building a multi-dimensional large language model capability framework provided by an embodiment of the present invention; in a feasible implementation manner, a multi-dimensional large language model capability framework is constructed according to cognitive dimension capabilities, domain dimension capabilities and task dimension capabilities; a trained multi-dimensional capability annotation model is obtained by training the multi-dimensional large language model capability framework; the trained multi-dimensional capability annotation model is applied to general scenario data screening based on diversity to obtain high-quality fine-tuning data in general scenarios; the trained multi-dimensional capability annotation model is applied to capability-oriented specific scenario data screening to obtain high-quality fine-tuning data in specific scenarios.

[0142] Among them, Figure 3The figure shows a schematic diagram of the cognitive dimension capability, task dimension capability and domain dimension capability architecture of a method for building a multi-dimensional large language model capability framework provided by an embodiment of the present invention; in a feasible implementation manner, the cognitive dimension capabilities defined in this application include: pattern recognition, concept abstraction, hypothesis generation, general sequence reasoning, quantitative reasoning, communication ability, mathematical performance, reading decoding, writing ability, naming function, associative flow, expression flow, vocabulary fluency, originality / creativity, idea fluency and problem perception / alternative solution fluency.

[0143] Among them, the task dimension capabilities defined in this application include: natural language reasoning, content rewriting, summary generation, classification, brainstorming, sentiment analysis, completion, generation, bias and fairness, word sense disambiguation, multiple-choice question answering, closed-book question answering, and extraction question answering.

[0144] Among them, the field dimension capabilities defined in this application include: linguistics, literature, multilingualism, tradition, art, sports, mass media, music, diet, health, biology, earth science, astronomy, chemistry, physics, mathematics, logic, economics, law, politics, education, sociology, agriculture, computer science, automation, electronics, engineering, programming, communication, religion, philosophy, ethics and history.

[0145] Based on the multimodal human cognitive ability of the CHC theoretical model, the language ability is screened out; based on the language ability, the domain-related abilities and non-core abilities of the large language model are removed to preliminarily obtain the cognitive dimension ability of the large language model; based on the preliminarily obtained cognitive dimension ability of the large language model, the inductive ability is refined into pattern recognition ability, concept abstraction ability and hypothesis generation ability, and finally the cognitive dimension ability of the large prediction model is defined; based on the FLASK domain classification system, the domain dimension ability of the large language model is defined; the task dimension ability of the large language model is defined; according to the cognitive dimension ability of the large language model and the domain dimension ability of the large language model, the task dimension ability of the large language model is defined. The multi-dimensional large language model capability framework is constructed by combining the task dimension capabilities of the large language model with the capability annotation training set; the capability annotation training set is obtained, and the GPT-4o model is used to perform fine-grained capability label annotation on each instruction in the capability annotation training set to obtain the annotated data set; according to the annotated data set, the Qwen2.5-7B-Base model is used to train the multi-dimensional large language model capability framework to obtain a trained multi-dimensional capability annotation model; the fine-tuning data of the large language model to be screened is obtained; the fine-tuning data of the large language model to be screened is input into the trained multi-dimensional capability annotation model to obtain high-quality fine-tuning data.

[0146] The present invention can be applied to diversity-based general scenario data screening to obtain high-quality fine-tuning data in general scenarios, or can be applied to capability-oriented specific scenario data screening to obtain high-quality fine-tuning data in specific scenarios; the application can improve the accuracy of data screening.

[0147] Figure 4 This is a block diagram of a device for building a multi-dimensional large language model capability framework according to an exemplary embodiment. The device is used in a method for building a multi-dimensional large language model capability framework. Figure 4 The device includes a first definition unit 410, a second definition unit 420, a construction unit 430, a labeling unit 440 and a screening unit 450. Among them:

[0148] The first definition unit 410 is used to screen out language ability based on the multimodal human cognitive ability of the CHC theoretical model; based on the language ability, remove the field-related ability and the non-core ability of the large language model, and preliminarily obtain the cognitive dimension ability of the large language model; based on the preliminarily obtained cognitive dimension ability of the large language model, refine the inductive ability into pattern recognition ability, concept abstraction ability and hypothesis generation ability, and finally define the cognitive dimension ability of the large prediction model;

[0149] The second definition unit 420 is used to define the domain dimension capability of the large language model based on the FLASK domain classification system; define the task dimension capability of the large language model;

[0150] A construction unit 430 is used to construct a multi-dimensional large language model capability framework according to the cognitive dimension capability of the large language model, the domain dimension capability of the large language model, and the task dimension capability of the large language model;

[0151] The labeling unit 440 is used to obtain a capability labeling training set, and use the GPT-4o model to label each instruction in the capability labeling training set with a fine-grained capability label to obtain a labeled data set;

[0152] The screening unit 450 is used to train the multi-dimensional large language model capability framework using the Qwen2.5-7B-Base model according to the annotated data set to obtain a trained multi-dimensional capability annotation model; obtain the fine-tuning data of the large language model to be screened; and input the fine-tuning data of the large language model to be screened into the trained multi-dimensional capability annotation model to obtain high-quality fine-tuning data.

[0153] Optionally, the multi-dimensional large language model capability framework is expressed by the following formula (1):

[0154] (1)

[0155] in, represents the multi-dimensional large language model capability framework; c represents cognitive ability; d represents domain; t represents task; represents the cognitive dimension; Represents the domain dimension; Represents the task dimension.

[0156] Optionally, the formula of the cognitive dimension is expressed by the following formula (2):

[0157] (2)

[0158] in, Indicates specific cognitive abilities; represents the total number of cognitive dimensions;

[0159] The formula of the domain dimension is expressed by the following formula (3):

[0160] (3)

[0161] in, Indicates a segmented area; Indicates the total number of domain dimensions;

[0162] The formula of the task dimension is expressed by the following formula (4):

[0163] (4)

[0164] in, Represents the defined task; Indicates the total number of task dimensions.

[0165] Optionally, the labeling unit 440 is used to:

[0166] According to the ability labeling training set, two cognitive dimension ability labels are labeled for each data through the GPT-4o model, one domain dimension ability label is labeled for each data through the GPT-4o model, and one task dimension ability label is labeled for each data through the GPT-4o model; by labeling the cognitive dimension ability, domain dimension ability and task dimension ability of each data, the labeled data set is obtained.

[0167] Optionally, the trained multi-dimensional capability labeling model includes: a trained cognitive dimension capability labeling model, a trained domain dimension capability labeling model and a trained task dimension capability labeling model.

[0168] Optionally, the screening unit 450 is used to:

[0169] Input the fine-tuning data of the large language model to be screened into the trained multi-dimensional ability annotation model, and obtain the annotated data pool by annotating the cognitive dimension ability, domain dimension ability, and task dimension ability;

[0170] Based on the annotated data pool, define all the composite capabilities in the annotated data pool; where the composite capabilities are the combination of cognitive dimension capabilities, domain dimension capabilities, and task dimension capabilities;

[0171] Fine-tuning data is screened based on all the composite capabilities in the annotated data pool to obtain high-quality fine-tuning data.

[0172] Optionally, the process of defining all composite capabilities in the labeled data pool is represented by the following formula (5):

[0173] (5)

[0174] in, Represents all composite capabilities in the annotated data pool; Represents the annotated data pool; A composite capability set representing a labeled data pool.

[0175] Based on the multimodal human cognitive ability of the CHC theoretical model, the language ability is screened out; based on the language ability, the domain-related abilities and non-core abilities of the large language model are removed to preliminarily obtain the cognitive dimension ability of the large language model; based on the preliminarily obtained cognitive dimension ability of the large language model, the inductive ability is refined into pattern recognition ability, concept abstraction ability and hypothesis generation ability, and finally the cognitive dimension ability of the large prediction model is defined; based on the FLASK domain classification system, the domain dimension ability of the large language model is defined; the task dimension ability of the large language model is defined; according to the cognitive dimension ability of the large language model and the domain dimension ability of the large language model, the task dimension ability of the large language model is defined. The multi-dimensional large language model capability framework is constructed by combining the task dimension capabilities of the large language model with the capability annotation training set; the capability annotation training set is obtained, and the GPT-4o model is used to perform fine-grained capability label annotation on each instruction in the capability annotation training set to obtain the annotated data set; according to the annotated data set, the Qwen2.5-7B-Base model is used to train the multi-dimensional large language model capability framework to obtain a trained multi-dimensional capability annotation model; the fine-tuning data of the large language model to be screened is obtained; the fine-tuning data of the large language model to be screened is input into the trained multi-dimensional capability annotation model to obtain high-quality fine-tuning data.

[0176] The present invention can be applied to diversity-based general scenario data screening to obtain high-quality fine-tuning data in general scenarios, or can be applied to capability-oriented specific scenario data screening to obtain high-quality fine-tuning data in specific scenarios; the application can improve the accuracy of data screening.

[0177] Figure 5 is a schematic diagram of a structure of a device for building a multi-dimensional large language model capability framework provided by an embodiment of the present invention, such as Figure 5 As shown, the equipment for building the multi-dimensional large language model capability framework may include the above Figure 4 The device for building a multi-dimensional large language model capability framework is shown. Optionally, the device 510 for building a multi-dimensional large language model capability framework may include a first processor 2001.

[0178] Optionally, the device 510 for building a multi-dimensional large language model capability framework may also include a memory 2002 and a transceiver 2003 .

[0179] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0180] Combine the following Figure 5 The components of the device 510 built with the multi-dimensional large language model capability framework are introduced in detail:

[0181] Among them, the first processor 2001 is the control center of the device 510 built with the multi-dimensional large language model capability framework, which can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0182] Optionally, the first processor 2001 can execute various functions of the device 510 built with the multi-dimensional large language model capability framework by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0183] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 5 CPU0 and CPU1 are shown in FIG.

[0184] In a specific implementation, as an embodiment, the device 510 for building a multi-dimensional large language model capability framework may also include multiple processors, such as Figure 5 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0185] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0186] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0187] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0188] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 5 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0189] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0190] It should be noted that Figure 5 The structure of the device 510 built with the multi-dimensional large language model capability framework shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0191] In addition, the technical effects of the device 510 for building a multi-dimensional large language model capability framework can refer to the technical effects of the method for building a multi-dimensional large language model capability framework described in the above method embodiment, and will not be repeated here.

[0192] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0193] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0194] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0195] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0196] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0197] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0198] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0199] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0200] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0201] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0202] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0203] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0204] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for building a multi-dimensional large language model capability framework, characterized in that: The method comprises: S1. Based on the multimodal human cognitive ability of the CHC theoretical model, the language ability is screened out; based on the language ability, the domain-related abilities and the non-core abilities of the large language model are removed to preliminarily obtain the cognitive dimension ability of the large language model; based on the preliminarily obtained cognitive dimension ability of the large language model, the inductive ability is refined into pattern recognition ability, concept abstraction ability and hypothesis generation ability, and finally the cognitive dimension ability of the large prediction model is defined; S2. Based on the FLASK domain classification system, define the domain dimension capabilities of the large language model; define the task dimension capabilities of the large language model; S3. Construct a multi-dimensional large language model capability framework based on the cognitive dimension capability of the large language model, the domain dimension capability of the large language model, and the task dimension capability of the large language model; S4. Obtain a capability labeling training set, use the GPT-4o model to perform fine-grained capability labeling on each instruction in the capability labeling training set, and obtain a labeled data set; S5. According to the annotated data set, the Qwen2.5-7B-Base model is used to train the multi-dimensional large language model capability framework to obtain a trained multi-dimensional capability annotation model; the fine-tuning data of the large language model to be screened is obtained; the fine-tuning data of the large language model to be screened is input into the trained multi-dimensional capability annotation model to obtain high-quality fine-tuning data.

2. The method for building a multi-dimensional large language model capability framework according to claim 1, characterized in that: The multi-dimensional large language model capability framework is expressed by the following formula (1): （1） in, represents the multi-dimensional large language model capability framework; c represents cognitive ability; d represents domain; t represents task; represents the cognitive dimension; Represents the domain dimension; Represents the task dimension.

3. The method for building a multi-dimensional large language model capability framework according to claim 2, characterized in that: The formula of the cognitive dimension is expressed by the following formula (2): （2） in, Indicates specific cognitive abilities; represents the total number of cognitive dimensions; The formula of the domain dimension is expressed by the following formula (3): （3） in, Indicates a segmented area; Indicates the total number of domain dimensions; The formula of the task dimension is expressed by the following formula (4): （4） in, Represents the defined task; Indicates the total number of task dimensions.

4. The method for building a multi-dimensional large language model capability framework according to claim 1, characterized in that: The S4 uses the GPT-4o model to perform fine-grained capability labeling on each instruction in the capability labeling training set to obtain a labeled data set, including: According to the ability labeling training set, two cognitive dimension ability labels are labeled for each data through the GPT-4o model, one domain dimension ability label is labeled for each data through the GPT-4o model, and one task dimension ability label is labeled for each data through the GPT-4o model; by labeling the cognitive dimension ability, domain dimension ability and task dimension ability of each data, the labeled data set is obtained.

5. The method for building a multi-dimensional large language model capability framework according to claim 1, characterized in that: The trained multi-dimensional capability labeling model includes: a trained cognitive dimension capability labeling model, a trained domain dimension capability labeling model and a trained task dimension capability labeling model.

6. The method for building a multi-dimensional large language model capability framework according to claim 1, characterized in that: The step of obtaining fine-tuning data of the large language model to be screened; Input the fine-tuning data of the large language model to be screened into the trained multi-dimensional capability annotation model to obtain high-quality fine-tuning data, including: Input the fine-tuning data of the large language model to be screened into the trained multi-dimensional ability annotation model, and obtain the annotated data pool by annotating the cognitive dimension ability, domain dimension ability, and task dimension ability; Based on the annotated data pool, define all the composite capabilities in the annotated data pool; where the composite capabilities are the combination of cognitive dimension capabilities, domain dimension capabilities, and task dimension capabilities; Fine-tuning data is screened based on all the composite capabilities in the annotated data pool to obtain high-quality fine-tuning data.

7. The method for building a multi-dimensional large language model capability framework according to claim 6, characterized in that: The process of defining all composite capabilities in the labeled data pool is expressed by the following formula (5): （5） in, Represents all composite capabilities in the annotation data pool; Represents the annotated data pool; A composite capability set representing a labeled data pool.

8. A device for building a multi-dimensional large language model capability framework, the device for building a multi-dimensional large language model capability framework is used to implement the method for building a multi-dimensional large language model capability framework as described in any one of claims 1-7, characterized in that: The device comprises: The first definition unit is used to screen out language ability based on the multimodal human cognitive ability of the CHC theoretical model; based on language ability, remove domain-related abilities and non-core abilities of the large language model to preliminarily obtain the cognitive dimension ability of the large language model; based on the preliminarily obtained cognitive dimension ability of the large language model, refine the inductive ability into pattern recognition ability, concept abstraction ability and hypothesis generation ability, and finally define the cognitive dimension ability of the large prediction model; The second definition unit is used to define the domain dimension capabilities of the large language model based on the FLASK domain classification system; define the task dimension capabilities of the large language model; A construction unit, used to construct a multi-dimensional large language model capability framework according to the cognitive dimension capability of the large language model, the domain dimension capability of the large language model, and the task dimension capability of the large language model; The labeling unit is used to obtain the capability labeling training set, and use the GPT-4o model to label each instruction in the capability labeling training set with a fine-grained capability label to obtain the labeled data set; The screening unit is used to train the multi-dimensional large language model capability framework using the Qwen2.5-7B-Base model according to the labeled data set to obtain a trained multi-dimensional capability annotation model; obtain the fine-tuning data of the large language model to be screened; and input the fine-tuning data of the large language model to be screened into the trained multi-dimensional capability annotation model to obtain high-quality fine-tuning data.

9. A device for building a multi-dimensional large language model capability framework, characterized in that: The equipment for building the multi-dimensional large language model capability framework includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images