Category preference setting method based on model evaluation index and related device

Abstract

The present application relates to a method for setting category preferences based on model evaluation indicators and related devices, the method comprising the following steps: detecting the model type and obtaining the feature dimension to be evaluated of the model; detecting the output structure of the model's logits, and creating an adjustment layer based on the detected output structure; obtaining the pre-entered preference parameters based on the feature dimension to be evaluated, and determining the adjustment coefficient based on the preference parameters, the adjustment coefficient being an element in the adjustment layer and corresponding to the position of the feature dimension; detecting the output layer of the model, and inserting the adjustment medium before the output layer, the output layer being a linear layer for generating evaluation results using the output function and the logits value operation. The method quickly sets the preference parameters of the adaptation indicator category according to the specific evaluation indicator requirements, can meet the customization requirements in diversified application scenarios, and reduce the time and labor cost required for customization while maintaining the overall performance of the model.

Description

Technical Field

[0001] The present application relates to the field of machine learning, and in particular to a method for setting category preferences based on model evaluation indicators and related devices. Background Art

[0002] In modern machine learning applications, customizing models to fit specific business needs and evaluation metrics is an increasingly important task. With the widespread application of machine learning techniques, various fields, from image recognition to natural language processing, have put forward specific requirements on the performance of models. In order to meet these requirements, personalized customization of models has become particularly critical. However, personalized adjustment of model weights, especially the weights of the logits layer during the inference phase of the model, faces several challenges.

[0003] Traditionally, this kind of personalized customization of models often means that the model training process needs to be manually modified or adjusted. This process is not only complex and time-consuming, but it is also difficult to ensure that the modifications made can meet the optimization requirements of specific indicators without adversely affecting other performance indicators of the model. In addition, these customization methods usually lack sufficient versatility, and they may only be applicable to specific fields or specific types of tasks. When faced with different application scenarios, this lack of versatility in customization limits the flexibility and wide applicability of the model.

[0004] In practical applications, a model may need to adapt to a variety of different usage scenarios. In each scenario, the output of the model (i.e., logits) may need to be adjusted to varying degrees to optimize specific evaluation metrics. These adjustments usually involve modifying the weights of the model's logits values. However, adjusting the logits weights is not a simple task. It requires a deep understanding of the working mechanism of the model and the ability to accurately match the specific logits that need to be adjusted. In addition, the situation becomes more complicated when such adjustments need to be migrated between multiple models. Each model may have its own unique architecture and output structure, which requires each model to be analyzed and matched separately, further increasing the complexity and time cost of customization. Summary of the invention

[0005] In order to quickly realize personalized customization of the model by performing a small amount of adjustment operations during the model inference stage, the present application provides a category preference setting method based on model evaluation indicators and related devices.

[0006] In the first aspect, the present application provides a method for setting category preference based on model evaluation indicators, which adopts the following technical solution:

[0007] A method for setting a category preference based on a model evaluation indicator comprises the following steps:

[0008] Detect the model type and obtain the feature dimensions of the model to be evaluated;

[0009] Detect the output structure of the model's logits and create an adjustment layer based on the detected output structure;

[0010] Acquire a pre-recorded preference parameter based on the feature dimension to be evaluated, and determine an adjustment coefficient based on the preference parameter, wherein the adjustment coefficient is an element in the adjustment layer and corresponds to the position of the feature dimension;

[0011] An output layer of the model is detected, and the adjustment medium is inserted before the output layer, wherein the output layer is a linear layer for generating an evaluation result using an output function and a logits value operation.

[0012] Optionally, the model type is a machine learning model, including at least a binary classification model, a multi-classification model, a sequence-to-sequence model and / or a sequence labeling model.

[0013] Optionally, the output structure of logits for binary classification models is a single value; the output structure of logits for multi-classification models is a vector whose length is equal to the number of categories, and each element represents the strength of the model's prediction for the corresponding category; the output structure of logits for multi-label classification models is a vector whose length is equal to the number of labels, and the model predicts the likelihood of each label independently; the output structure of logits for sequence-to-sequence models is a matrix, where each row represents the output vector for each time step in the sequence, and the length of each vector is equal to the size of the vocabulary.

[0014] Optionally, the structure of the adjustment layer is a matrix, and the number of columns of the matrix corresponds to the number of columns of the output structure of logits, and the number of rows of the matrix is ​​1 or matches the maximum length of the sequence.

[0015] Optionally, the elements of each row of the adjustment layer are used to multiply the logits corresponding to the feature dimension output, and the operation results are added to the corresponding positions of a blank matrix of the same size as the adjustment layer.

[0016] Optionally, when the row element of the adjustment layer is greater than 1, it is multiplied by the logits corresponding to the feature dimension output to increase the possibility of the model in the preference category and improve the recall rate; when the row element of the adjustment layer is less than 1, it is multiplied by the logits corresponding to the feature dimension output to suppress the output of the model in the preference category and improve the accuracy.

[0017] In the second aspect, the present application provides a category preference setting system based on model evaluation indicators, which adopts the following technical solutions:

[0018] A category preference setting system based on model evaluation indicators, comprising:

[0019] Model detection module, used to detect the model type and obtain the feature dimensions of the model to be evaluated;

[0020] The adjustment layer creation module is used to detect the output structure of the model's logits and create an adjustment layer based on the detected output structure;

[0021] An adjustment layer input module, used to obtain a pre-entered preference parameter based on the feature dimension to be evaluated, and determine an adjustment coefficient based on the preference parameter, wherein the adjustment coefficient is an element in the adjustment layer and corresponds to the position of the feature dimension;

[0022] The adjustment layer insertion module is used to detect the output layer of the model and insert the adjustment medium before the output layer, wherein the output layer is a linear layer used to generate an evaluation result by using an output function and a logits value operation.

[0023] In a third aspect, the present application provides a computer device, which adopts the following technical solution:

[0024] A computer device comprising:

[0025] one or more processors;

[0026] Memory;

[0027] One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs are configured to: execute the above-mentioned category preference setting method based on model evaluation indicators.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0029] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute the above method.

[0030] The storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement: the category preference setting method based on the model evaluation indicator as described above.

[0031] In summary, the present application includes at least one of the following beneficial technical effects: adjusting the logits weights in the model inference stage through simplified operations. This method does not require changing the original training process, and only requires setting the threshold in the inference stage to take effect. It is flexible and easy to implement. This method allows the weights of the model output to be automatically set and adjusted according to different evaluation indicator requirements, thereby achieving fast and flexible model personalization. In this way, the model can be quickly adjusted in different application scenarios, and the weight adjustment can be easily migrated and applied in multi-model collaborative scenarios, which greatly saves the time and resource consumption of model adjustment and improves the efficiency and applicability of model customization. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a method for setting a category preference based on a model evaluation index according to an embodiment of the present invention is shown.

[0033] Figure 2 A schematic diagram of a category preference setting system based on model evaluation indicators according to an embodiment of the present invention is shown.

[0034] Figure 3 A schematic diagram of a computer device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0035] The present application is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the inventive concept. As part of this specification, some of the drawings in the drawings of the present disclosure represent structures and devices in the form of block diagrams to avoid making the disclosed principles complicated and difficult to understand. For the sake of clarity, not all features of the actual implementation are necessary to be described. In addition, the language used in this disclosure has been selected primarily for readability and instructional purposes, and may not be selected to delineate or limit the subject matter of the present invention, thereby resorting to the necessary claims to determine such inventive subject matter. References to "a specific implementation" or "specific implementations" in this disclosure mean that the specific features, structures or characteristics described in conjunction with the specific implementation are included in at least one specific implementation, and multiple references to "a specific implementation" or "specific implementations" should not be understood as necessarily referring to the same specific implementation.

[0037] Unless expressly limited, the terms "a", "an", and "the" are not intended to refer to a singular entity, but rather to include general categories of which specific examples may be used for illustration. Thus, the use of the terms "a" or "an" may mean any number of at least one, including "one", "one or more", "at least one", and "one or more than one". The term "or" means any of the alternatives and any combination of the alternatives, including all alternatives, unless the alternatives are expressly indicated to be mutually exclusive. The phrase "at least one of" when combined with a list of items refers to a single item in the list or any combination of the items in the list. The phrase does not require all of the listed items unless expressly limited to that.

[0038] Reference Figure 1 , the category preference setting method based on model evaluation indicators includes the following steps:

[0039] S1. Detect the model type and obtain the feature dimensions of the model to be evaluated.

[0040] At present, when technicians first come into contact with a new model, they usually determine the type of model by consulting the documentation or analyzing the source code. For example, if the model comes from an open source library or framework, you can consult the official documentation of the model. The documentation usually provides a detailed description of the model, including its type (such as CNN, RNN, Transformer, etc.) and purpose (such as classification, regression, sequence prediction, etc.). If you have access rights, directly viewing the source code of the model is an effective way to obtain information about the model type and feature dimensions. In the source code, you can find the model definition section, which usually includes the type of layer and configuration parameters.

[0041] In this solution, in order to improve recognition efficiency, an automated approach is used to detect the model type and obtain the feature dimensions to be evaluated using a programming interface. This usually involves using framework-specific functions or methods to query the attributes of the model. Here, examples are given based on two deep learning frameworks, PyTorch and TensorFlow (Keras).

[0042] In PyTorch, you can detect the model type and obtain feature dimensions by directly accessing the model's attributes and structure.

[0043] Detection model type:

[0044] import torch.nn as nn

[0045] #Assume model is your PyTorch model instance

[0046] model=...

[0047] #Get the class name of the model

[0048] model_type=type(model).__name__

[0049] print(f"Model type:{model_type}")

[0050] Then, based on the type of model detected, the feature dimension is determined. For example, for a convolutional neural network (CNN), the feature dimension usually refers to the number of channels of the input image, which can be obtained by inspecting the first convolutional layer of the model:

[0051] #Assume that the first layer of the model is a convolutional layer

[0052] first_conv_layer=next(model.children())

[0053] input_channels=first_conv_layer.in_channels

[0054] print(f"Input feature dimension(channels):{input_channels}").

[0055] For example, when using TensorFlow (especially its high-level API Keras), you can detect the model type and obtain the feature dimension in a similar way.

[0056] Detection model type:

[0057] from tensorflow.keras.models import Sequential,Model

[0058] #Assume model is your Keras model instance

[0059] model=...

[0060] #Detect model type

[0061] ifisinstance(model,Sequential):

[0062] model_type = 'Sequential'

[0063] elifisinstance(model,Model):

[0064] model_type='FunctionalAPI Model'

[0065] else:

[0066] model_type = 'Unknown'

[0067] print(f"Model type:{model_type}")

[0068] Get feature dimensions:

[0069] In Keras, you can get the feature dimensions by inspecting the input layer of the model. This works for both sequence models and models built using the FunctionalAPI:

[0070] #Get the shape of the model input layer

[0071] input_shape=model.input_shape

[0072] #input_shape is a tuple, for example, for image data it may be (batch size, height, width, number of channels)

[0073] #Feature dimensions usually refer to the part other than batch size

[0074] feature_dims = input_shape[1:] # ignore batch size

[0075] print(f"Feature dimensions:{feature_dims}")

[0076] By using these programming interfaces, you can easily and automatically obtain the type and feature dimension information of the model in a script or application, so as to further analyze and process the model.

[0077] S2. Detect the output structure of the model's logits and create an adjustment layer based on the detected output structure.

[0078] The logits output structure of a model usually depends on the requirements of a specific task and the model design. In machine learning and deep learning models, logits are the output of the last linear layer (i.e., the layer before the output layer), and are the raw prediction values ​​that have not yet been converted into probabilities by applying activation functions such as softmax or sigmoid. Depending on the application scenario, the output structure of logits can take on a variety of forms:

[0079] Binary classification task: For binary classification problems, the output structure of logits is usually a single value (i.e. one dimension) that represents the prediction strength of belonging to the positive class. Afterwards, this logit is usually converted into a probability value through the sigmoid function, indicating the probability that the sample belongs to the positive class.

[0080] Multi-classification task: In multi-classification problems, the output structure of logits is a vector whose length is equal to the number of categories. Each element represents the strength of the model's prediction of the corresponding category. These logits values ​​will be converted into a probability distribution through the softmax function, and each element represents the probability that the sample belongs to the corresponding category.

[0081] Multi-label classification task: In multi-label classification tasks, the model needs to predict whether each category appears independently, rather than selecting one from multiple categories. In this case, the output structure of logits is a vector whose length is equal to the number of labels, and the model predicts the probability of each label to appear independently. Each logit is converted into a probability through the sigmoid function, indicating the probability of the corresponding label appearing.

[0082] Sequence to sequence tasks: For sequence to sequence (Seq2Seq) tasks, such as machine translation or text generation, the output structure of logits is a matrix where each row represents the output vector for each time step in the sequence, and the length of each vector is equal to the size of the vocabulary. Each row of logits is converted into a probability distribution through the softmax function, representing the probability of generating each possible word at each time step.

[0083] For example, in an image classification task with 10 categories, the logits output structure of the model will be a 10-dimensional vector. The value of each dimension represents the strength of the model's prediction that the image belongs to the corresponding category.

[0084] For example, taking text sentiment analysis as an example, in a binary sentiment analysis task (positive or negative sentiment), the output structure of logits may be a single value, which is converted into a numerical value representing the probability of positive sentiment through the sigmoid function.

[0085] For example, taking machine translation as an example, assuming the vocabulary size is 10,000, the model outputs a 10,000-dimensional vector for each generated word, where each dimension corresponds to a word in the vocabulary, indicating the probability that the next word is each word in the vocabulary.

[0086] It should be noted that the structure of the adjustment layer is a matrix, and the number of columns of the matrix corresponds to the number of columns of the output structure of logits, and the number of rows of the matrix is ​​1 or matches the maximum length of the sequence.

[0087] Here we take the model architecture of using a pre-trained model plus a conditional random field as an example. Here we use the pre-trained model as the basis and output a vector representation of a batch of text with a dimension of [batch_size, max_sequence_length, hidden_size]. Among them, batch_size is the batch size of the model input sequence, max_sequence_length is the longest text of the model input batch, and hidden_size is the dimension of the hidden state output by the model.

[0088] Perform a linear transformation on the text vector output by the pre-trained model, map the hidden state to the number of tags, and obtain an emission matrix with the dimension [batch_size, max_sequence_length, tag_num]. Among them, batch_size is the batch size of the model input sequence, max_sequence_length is the longest text of the model input batch, and tag_num is the number of all tags in the model. The emission matrix provides the logits value of each token position and tag combination to measure the possibility that the position belongs to the tag.

[0089] Then create an adjustment matrix with the same shape as the emission matrix, with dimensions [batch_size, max_sequence_length, tag_num]. Initialize it to a matrix of all 1s, indicating that logits are not adjusted by default.

[0090] For each indicator category label (B-category name, I-category name, E-category name), determine its modification coefficient according to the indicator to be improved (precision or recall):

[0091] High precision: The corresponding coefficient is less than 1, for example 0.8.

[0092] High recall: The corresponding coefficient is greater than 1, for example 1.2.

[0093] According to the label category and the specified modification coefficient, the value of the corresponding label position in the adjustment matrix is ​​updated. For example, if the recall rate of a specific entity category needs to be improved, the coefficient of the corresponding position of the B, I, and E labels of that category is set to a value greater than 1.

[0094] Then, the adjustment matrix is ​​multiplied element by element with the original emission matrix to obtain the adjusted emission matrix. This step can be easily implemented using a matrix operation library (such as NumPy, PyTorch, TensorFlow, etc.).

[0095] The adjusted emission matrix is ​​used as the input of the conditional random field (CRF) model. The CRF model will use these adjusted logits to infer the best label sequence for each token. The CRF model performs reasoning based on the adjusted emission matrix and outputs the best label for each token to obtain the final sequence labeling result, whose dimension is [batch_size, sequence_length].

[0096] S3. Obtaining pre-recorded preference parameters based on the feature dimension to be evaluated, and determining an adjustment coefficient based on the preference parameters, wherein the adjustment coefficient is an element in the adjustment layer and corresponds to the position of the feature dimension.

[0097] The design of preference parameters is a key component of personalized adjustment of machine learning models, which allows the model to be flexibly adjusted according to different application scenarios and evaluation indicator requirements. The purpose of pre-entering preference parameters in the system is to define the importance or preference of each feature dimension in advance, and these parameters can be set according to the needs of the actual application scenario. The advantage of this approach is that once these parameters are determined and stored in the database, they can be reused to provide guidance for different model instances, thereby achieving rapid model customization and adjustment.

[0098] In practical applications, preference parameters can be retrieved from the database according to the feature dimensions to be processed by the model. The advantage of this method is that when the model needs to be applied to new scenarios or faces different task requirements, the behavior of the model can be adjusted by simply retrieving the corresponding preference parameters without complicated reconfiguration and training.

[0099] As shown in S2, the shape of the adjustment matrix is ​​the same as that of the emission matrix, and its dimension is [batch_size, max_sequence_length, tag_num]. The adjustment coefficient is an element in the adjustment layer and corresponds to the position of the feature dimension, that is, for the label of each indicator category (B-category name, I-category name, E-category name), the modification coefficient is determined according to the indicator to be improved (precision or recall rate) and filled in the corresponding position.

[0100] S4. Detect the output layer of the model and insert the adjustment medium before the output layer, wherein the output layer is a linear layer for generating an evaluation result using an output function and a logits value operation.

[0101] Detecting the output layer of a model, especially when dealing with deep learning models with output functions and logits values, involves understanding the model architecture and identifying its last linear layer (i.e., the output layer). The output layer is the last layer of the network and directly affects the final prediction results of the model. It is usually responsible for converting the high-level feature representations of the previous layers into an output format that is meaningful to the specific task, such as class probabilities in classification problems. Most modern deep learning frameworks (such as TensorFlow, PyTorch) provide APIs to query and manipulate different parts of the model. You can use these APIs to get the hierarchical structure of the model and identify the last linear layer. In addition, since the dimensions of the output layer are usually aligned with the objectives of the task (such as the number of categories for classification tasks), this can be used as a way to identify the output layer. In addition, in many cases, the output layer uses a specific activation function (such as softmax for multi-classification tasks), which can also help identify the output layer.

[0102] Embodiment 1:

[0103] Step 1: Preparation

[0104] First, a pre-trained model is used as a basis, which can output vector representations of each batch of text. These vector representations are mapped to a dimension corresponding to the number of labels through a linear transformation layer to generate an emission matrix. Each element of the emission matrix represents the logits value of each token in the sequence on each label, that is, the possibility that the position belongs to the label.

[0105] Step 2: Define the adjustment factor

[0106] Based on the preference of the model indicator category, the adjustment coefficient is calculated. This involves specifying a coefficient for each label in the sequence annotation (such as B-class name, I-class name, E-class name, O), which will vary depending on whether precision or recall needs to be improved:

[0107] For high accuracy requirements, a coefficient less than 1 is selected to reduce the prediction frequency of specific category labels and reduce misjudgment.

[0108] For high recall requirements, a coefficient greater than 1 is selected to increase the prediction frequency of a specific class label and capture more positive examples.

[0109] Step 3: Apply the adjustment factor

[0110] Create a resize layer matrix of the same dimensions as the emission matrix and apply the resize coefficients for each label to the corresponding logits. This can be done by constructing a matrix containing the resize coefficients for each label and performing element-wise multiplication of this matrix with the emission matrix.

[0111] Step 4: Generate the adjusted emission matrix

[0112] Multiply the adjustment layer matrix with the original emission matrix to generate an adjusted emission matrix. This adjusted matrix will be used as the input of the conditional random field model, which reflects the label probabilities adjusted according to specific preferences.

[0113] Step 5: Model Inference

[0114] The adjusted emission matrix is ​​input into the conditional random field (CRF) model. The CRF model infers the best label sequence for each token based on the adjusted emission matrix. Finally, the model outputs the label prediction for each sequence, whose dimension is [batch_size, sequence_length], where each prediction has taken into account a specific preference adjustment.

[0115] Embodiment 2:

[0116] Taking the text classification model as an example, the specific steps are as follows:

[0117] Step 1: Determine the text classification model architecture

[0118] 1. Determine the text classification model architecture:

[0119] 1.1 Model architecture using pre-trained model, fully connected layer, and softmax layer.

[0120] 1.2 Using the pre-trained model BERT as a basis, we usually use the [CLS] tag output by the BERT model as the representation of the entire text. Extract a batch of [CLS]-tagged text vector representations from the output of BERT, with dimensions [batch_size, hidden_size]. Where batch_size is the batch size of the model input sequence, and hidden_size is the dimension of the hidden state output by the model. That is, for each text in each batch, a vector representing the entire text will be obtained.

[0121] 1.3 Input the vector represented by the extracted [CLS] tag into the fully connected layer, and the dimension becomes [batch_size, num_classes], where batch_size is the batch size of the model input sequence and num_classes is the number of categories of the model.

[0122] Step 2: Calculate the modification factor

[0123] 2. Calculate the modification coefficient for the preferred model indicator category:

[0124] 2.1 If a certain indicator category requires high precision, the logits value of the feature dimension corresponding to the indicator is multiplied by a coefficient less than 1 to suppress the model's output on the label of the preferred category and improve precision.

[0125] 2.2 If a certain indicator category requires a high recall rate, multiply the logits value of the feature dimension corresponding to the indicator by a coefficient greater than 1 to increase the possibility of the model outputting the label of the preference category and improve the recall rate.

[0126] 2.3 If there is no special requirement, the logits value of the indicator corresponding to the feature dimension remains unchanged and is not modified.

[0127] Step 3: Apply the modification factor

[0128] Create a resized matrix (or resized vector, since this is resized for each class in a particular batch) of the same dimension as logits. For each class, choose the resized factor based on its needs and multiply it by the original logits value.

[0129] Step 4: Model Inference

[0130] Determine the indicator category probability matrix: Input the modified feature vector matrix into the softmax layer to obtain the indicator category probability matrix corresponding to the text, with a dimension of [batch_size, num_classes].

[0131] The modified class probability matrix is ​​used for model inference to obtain output results adjusted for preference requirements.

[0132] Step 5: Model Evaluation

[0133] Evaluate the performance of the adjusted model using standard evaluation metrics such as precision, recall, F1 score, etc. By comparing the performance before and after the adjustment, you can verify whether the preference adjustment has achieved the desired effect.

[0134] Embodiment 3:

[0135] For the sequence labeling model, if the indicator category [charging speed] has a high-precision preference requirement, the logits value corresponding to the label [B-charging speed, I-charging speed, E-charging speed] corresponding to the indicator category is multiplied by a coefficient less than 1; if the indicator category [charging speed] has a high recall preference requirement, the logits value corresponding to the label [B-charging speed, I-charging speed, E-charging speed] corresponding to the indicator category is multiplied by a coefficient greater than 1. For the text classification model, if the indicator category [positive] has a high-precision preference requirement, the logits value corresponding to the indicator category [positive] is multiplied by a coefficient less than 1; if the indicator category [positive] has a high recall preference requirement, the logits value corresponding to the indicator category [positive] is multiplied by a coefficient greater than 1.

[0136] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0137] In one embodiment, a system for setting a category preference based on a model evaluation index is provided, and the system for setting a category preference based on a model evaluation index corresponds one-to-one to the method for setting a category preference based on a model evaluation index in the above embodiment. Figure 2 As shown, the category preference setting system based on model evaluation indicators includes a model detection module, an adjustment layer creation module, an adjustment layer input module and an adjustment layer insertion module. The detailed description of each functional module is as follows:

[0138] The model detection module is used to detect the model type and obtain the feature dimensions of the model to be evaluated.

[0139] The adjustment layer creation module is used to detect the output structure of the logits of the model and create an adjustment layer based on the detected output structure.

[0140] The adjustment layer input module is used to obtain pre-entered preference parameters based on the feature dimension to be evaluated, and determine the adjustment coefficient based on the preference parameters, wherein the adjustment coefficient is an element in the adjustment layer and corresponds to the position of the feature dimension.

[0141] The adjustment layer insertion module is used to detect the output layer of the model and insert the adjustment medium before the output layer, wherein the output layer is a linear layer used to generate an evaluation result by using an output function and a logits value operation.

[0142] For the specific limitations of the category preference setting system based on model evaluation indicators, please refer to the limitations of the category preference setting method based on model evaluation indicators in the above text, which will not be repeated here. Each module in the above-mentioned category preference setting system based on model evaluation indicators can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0143] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for data related to the category preference setting method based on the model evaluation index. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a category preference setting method based on the model evaluation index is implemented.

[0144] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for setting a category preference based on a model evaluation indicator in the above embodiment is implemented, for example Figure 1 Alternatively, when the processor executes the computer program, the functions of each module / unit of the category preference setting system based on the model evaluation index in the above embodiment are realized. To avoid repetition, they are not described here.

[0145] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for setting a category preference based on a model evaluation indicator in the above embodiment is implemented, for example Figure 1 Alternatively, when the computer program is executed by the processor, the functions of each module / unit in the category preference setting system based on the model evaluation index in the above device embodiment are realized. To avoid repetition, it will not be described here.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0147] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0148] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for setting category preference based on model evaluation indicators, characterized in that: The following steps are involved: Detecting the model type and obtaining the feature dimension of the model to be evaluated; wherein the feature dimension is a feature dimension of text or image; The output structure of the logits of the model is detected, and an adjustment layer is created based on the detected output structure; wherein the structure of the adjustment layer is a matrix, and the number of columns of the matrix corresponds to the number of columns of the output structure of logits, and the number of rows of the matrix is ​​1 or matches the maximum length of the sequence; the elements of each row of the adjustment layer are used to multiply the logits corresponding to the output of the feature dimension, and the operation result is added to the corresponding position of a blank matrix of the same size as the adjustment layer; the output structure corresponds to the probability of a possible word in the text or the probability that the image belongs to the corresponding category; Based on the feature dimension to be evaluated, the pre-entered preference parameters are obtained, and the adjustment coefficients are determined based on the preference parameters, wherein the adjustment coefficients are elements in the adjustment layer and correspond to the position of the feature dimension; when the row elements of the adjustment layer are greater than 1, they are multiplied with the logits output corresponding to the feature dimension to improve the possibility of the model in the preference category and improve the recall rate; when the row elements of the adjustment layer are less than 1, they are multiplied with the logits output corresponding to the feature dimension to suppress the output of the model in the preference category and improve the accuracy; An output layer of the model is detected, and the adjustment medium is inserted before the output layer, wherein the output layer is a linear layer for generating an evaluation result using an output function and a logits value operation.

2. The method for setting category preference based on model evaluation indicators according to claim 1, characterized in that: The model type is a machine learning model, which includes at least a binary classification model, a multi-classification model, a sequence-to-sequence model and / or a sequence labeling model.

3. The method for setting category preference based on model evaluation index according to claim 2, characterized in that: The output structure of logits for a binary classification model is a single value; the output structure of logits for a multi-classification model is a vector with a length equal to the number of categories, and each element represents the strength of the model's prediction of the corresponding category; the output structure of logits for a multi-label classification model is a vector with a length equal to the number of labels, and the model independently predicts the likelihood of each label appearing; the output structure of logits for a sequence-to-sequence model is a matrix, in which each row represents the output vector for each time step in the sequence, and the length of each vector is equal to the size of the vocabulary.

4. A category preference setting system based on model evaluation indicators, characterized in that: A method for setting a category preference based on a model evaluation indicator as claimed in any one of claims 1 to 3, comprising: Model detection module, used to detect the model type and obtain the feature dimensions of the model to be evaluated; The adjustment layer creation module is used to detect the output structure of the model's logits and create an adjustment layer based on the detected output structure; An adjustment layer input module, used to obtain a pre-entered preference parameter based on the feature dimension to be evaluated, and determine an adjustment coefficient based on the preference parameter, wherein the adjustment coefficient is an element in the adjustment layer and corresponds to the position of the feature dimension; The adjustment layer insertion module is used to detect the output layer of the model and insert the adjustment medium before the output layer, wherein the output layer is a linear layer used to generate an evaluation result by using an output function and a logits value operation.

5. A computer device, characterized in that: It includes: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs are configured to: execute the category preference setting method based on model evaluation indicators according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement: the category preference setting method based on model evaluation indicators as described in any one of claims 1 to 3.

Images