A counterfactual sample generation method and model adjustment method, device and medium

Abstract

Embodiments of the present invention disclose a method for generating counterfactual samples and a model adjustment method, device, and medium. The method includes: constructing a mirror network model symmetrical to the original machine learning network model based on the original machine learning network model to generate a counterfactual network model; performing model training and testing on the original machine learning network model using a test sample set to obtain test results; and using the target samples that were incorrectly identified in the test results as input to the counterfactual network model to obtain counterfactual samples output by the counterfactual network model. The test results of the testing phase can be automatically fed back to the learning phase, and counterfactual samples can be generated to expand the learning samples, solving the problem of small samples. Furthermore, the model can be automatically corrected without relying on human intervention, improving the recognition accuracy of the model and having universality.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a counterfactual sample generation method and model adjustment method, device, and medium. Background Art

[0002] The training process of a machine learning model is typically divided into a learning phase and a testing phase. The testing phase primarily tests the recognition accuracy of the currently learned model. The learning phase uses training samples, while the testing phase uses test samples. These samples are strictly isolated from the training samples, preventing the learning experience from the testing phase from being directly fed back into the learning phase. Furthermore, when the number of training samples is small, the model struggles to learn, exhibits significant bias, and lacks robustness, resulting in low machine learning recognition accuracy.

[0003] Existing techniques typically optimize machine learning models by manually analyzing test results from the testing phase. For example, data augmentation methods are used to increase the number of training samples and reduce model bias. However, existing approaches rely on extensive manual analysis, and some operations are difficult and not universally applicable. Summary of the Invention

[0004] The embodiments of the present invention provide a counterfactual sample generation method and a model adjustment method, device and medium, which can automatically feed back the test results of the testing phase to the learning phase, generate counterfactual samples to expand the learning samples, and solve the problem of small samples.

[0005] In a first aspect, an embodiment of the present invention provides a method for generating counterfactual samples, the method comprising:

[0006] Based on the original machine learning network model, construct a mirror network model that is symmetrical to the original machine learning network model to generate a counterfactual network model;

[0007] Performing model training and testing on the original machine learning network model using a test sample set, and obtaining test results;

[0008] The target sample that is incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model.

[0009] In a second aspect, an embodiment of the present invention further provides a counterfactual network model adjustment method, comprising:

[0010] Generate a counterfactual sample using the counterfactual sample generation method provided by any embodiment of the present invention;

[0011] Adding the counterfactual sample to the original training sample set to generate a target training sample set, and using the target training sample set to adjust the counterfactual network model provided by any embodiment of the present invention;

[0012] Performing model training and testing on the original machine learning network model in the counterfactual network model using a test sample set to obtain a test result;

[0013] If the test result does not meet the preset model training conditions, the target sample that is incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model; and the operation of adding the counterfactual sample to the original training sample set to generate the target training sample set is returned until the test result meets the preset model training conditions and the model adjustment is stopped.

[0014] In a third aspect, an embodiment of the present invention further provides a computer device, including a processor and a memory, wherein the memory is configured to store instructions, and when the instructions are executed, the processor performs the following operations:

[0015] Based on the original machine learning network model, construct a mirror network model that is symmetrical to the original machine learning network model to generate a counterfactual network model;

[0016] Performing model training and testing on the original machine learning network model using a test sample set, and obtaining test results;

[0017] The target sample that is incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer device, comprising a processor and a memory, wherein the memory is configured to store instructions, and when the instructions are executed, the processor performs the following operations:

[0019] Generate a counterfactual sample using the counterfactual sample generation method provided by any embodiment of the present invention;

[0020] Adding the counterfactual sample to the original training sample set to generate a target training sample set, and using the target training sample set to adjust the counterfactual network model provided by any embodiment of the present invention;

[0021] Performing model training and testing on the original machine learning network model in the counterfactual network model using a test sample set to obtain a test result;

[0022] If the test result does not meet the preset model training conditions, the target sample that is incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model; and the operation of adding the counterfactual sample to the original training sample set to generate the target training sample set is returned until the test result meets the preset model training conditions and the model adjustment is stopped.

[0023] In a fifth aspect, an embodiment of the present invention further provides a storage medium for storing instructions for executing:

[0024] Based on the original machine learning network model, construct a mirror network model that is symmetrical to the original machine learning network model to generate a counterfactual network model;

[0025] Performing model training and testing on the original machine learning network model using a test sample set, and obtaining test results;

[0026] Using the target sample that is incorrectly identified in the test result as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model;

[0027] or,

[0028] Generate a counterfactual sample using the counterfactual sample generation method provided by any embodiment of the present invention;

[0029] Adding the counterfactual sample to the original training sample set to generate a target training sample set, and using the target training sample set to adjust the counterfactual network model provided by any embodiment of the present invention;

[0030] Performing model training and testing on the original machine learning network model in the counterfactual network model using a test sample set to obtain a test result;

[0031] If the test result does not meet the preset model training conditions, the target sample that is incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model; and the operation of adding the counterfactual sample to the original training sample set to generate the target training sample set is returned until the test result meets the preset model training conditions and the model adjustment is stopped.

[0032] The technical solution of the embodiment of the present invention generates a counterfactual network model by constructing a mirror network model symmetrical to the original machine learning network model based on the original machine learning network model; training and testing the original machine learning network model using a test sample set to obtain test results; and using the target samples that were incorrectly identified in the test results as input to the counterfactual network model to obtain counterfactual samples output by the counterfactual network model. This solves the problem of small sample sizes in machine learning model training, automatically feeds test results from the testing phase back to the learning phase, generates counterfactual samples to expand the learning samples, and can automatically correct the model without relying on human intervention, improving the model's recognition accuracy and achieving universal results. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a generation flow chart of a machine learning model in the prior art;

[0034] Figure 2a This is a flowchart of a counterfactual sample generation method provided by Example 1 of the present invention;

[0035] Figure 2b This is a flowchart of another method for generating counterfactual samples provided in Example 1 of the present invention;

[0036] Figure 2c This is a flowchart of generating counterfactual samples based on a PIM model provided by the first embodiment of the present invention;

[0037] Figure 3 This is a flowchart of a counterfactual sample generation method provided by Example 2 of the present invention;

[0038] Figure 4 This is a flowchart of a counterfactual network model adjustment method provided by Example 3 of the present invention;

[0039] Figure 5 Schematic diagram of the structure of a counterfactual sample generation device provided by the fourth embodiment of the present invention;

[0040] Figure 6 Schematic diagram of the structure of a counterfactual network model adjustment device provided in Embodiment 5 of the present invention;

[0041] Figure 7 This is a structural diagram of a computer device provided in Example 6 of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0043] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the various operations (or steps) as sequential processes, many of the operations therein can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the various operations can be rearranged. The process can be terminated when its operations are completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0044] The term "counterfactual" used in this article can be a potential outcome in the development of a matter. For example, a matter may have multiple possibilities during its development, but in its current state it only has one or more of these possibilities; "counterfactual" can be understood as other possible future outcomes that may be different from the current state. For example, under natural conditions, there are always roosters crowing before sunrise. If this observation sample is given to the model for learning, the model will predict that the sun will rise when the roosters crow; but in fact, the crowing of roosters is not the reason for the sun to rise. Counterfactual learning will introduce a hypothesis that does not exist in reality: if all roosters do not crow, will the sun rise?

[0045] The term "counterfactual sample" used in this article refers to a counterfactual sample derived from the knowledge gained through studying the features of the currently learned sample. For example, if the color feature of an apple is cyan, then based on this knowledge, red apples or yellow apples can be derived. These two counterfactual samples are, relative to the green apple, counterfactual samples.

[0046] The term "normal net" used in this article may be a machine learning model in the prior art. For example, Figure 1 This is a flow chart for generating a machine learning model in the prior art. Figure 1As shown, the traditional machine learning model in the prior art can be generated through three steps: model learning, model testing, and model learning optimization. Specifically, model learning can be to use the training sample set to learn the model parameters, and update the parameters through continuous forward and backward calculation iterations until the loss function value is completely converged. Model testing can be to use the test sample set to test the learned model in stages during the model training process, and detect the model learning effect through indicators such as accuracy. Model learning optimization can be to manually analyze the test results, and according to the conclusions of the analysis, adjust the data set and retrain the model until the model effect meets the requirements and the model training is completed. The "original machine learning network model" in this article can be compared with Figure 1 The machine learning models in the paper have the same or similar network structures. However, the “original machine learning network model” in this paper can be compared with Figure 1 The model parameter determination methods of the machine learning models in are the same or different.

[0047] The term "mirror network model" used herein may be a mirror net. Specifically, a mirror net may be a network model that is structurally symmetrical to a normal net. For example, the mirror net may be structurally symmetrical to the normal net about the center of the feature vector output layer (feat layer). The input of the mirror net may be the feature vector output by the normal net; the output of the mirror net may be a counterfactual sample based on the feature vector.

[0048] The term "counterfactual network model" used in this article refers to a network model composed of an "original machine learning network model" and a "mirror network model." On the one hand, the "counterfactual network model" can output "counterfactual samples" through the "mirror network model"; on the other hand, the "counterfactual network model" can be used to test the model on a test sample set using the "original machine learning network model."

[0049] Example 1

[0050] Figure 2a This is a flowchart of a counterfactual sample generation method provided in Example 1 of the present invention. This embodiment is applicable to the situation where a small sample is expanded in the training of a machine learning model to generate counterfactual samples to adjust the machine learning model and thereby improve the recognition accuracy of the model. The method can be executed by a counterfactual sample generation device, which can be implemented in software and / or hardware. The device can be integrated into a computer, such as Figure 2a As shown, the method specifically includes:

[0051] Step 210: Based on the original machine learning network model, a mirror network model symmetrical to the original machine learning network model is constructed to generate a counterfactual network model.

[0052] Among them, the original machine learning network model can be any network model. The method of the embodiment of the present invention is universal and is not limited to specific data, specific framework or specific model structure. Exemplarily, the embodiment of the present invention is explained by taking the Plant Identification Model (PIM) as an example. Specifically, the original machine learning network model in the embodiment of the present invention can be a PIM model. The PIM model can take plant photos as input and output the category to which the plant belongs. The PIM model can use a mainstream classification network as a backbone. Exemplarily, the PIM model can use the AlexNet network as a backbone.

[0053] For example, in a specific application, 10,000 plant photos can be collected as samples and put into a sample library (PIM-Database). The size of the plant photos can be set according to needs, such as a size of 224×224×3. Plant photos can have species labels. For example, professional personnel can mark plant photos with species labels. Specifically, there can be 6 plant species labels. The samples can be randomly divided into training sample sets and test sample sets according to a certain ratio. For example, the ratio of the number of samples in the training sample set to the test sample set is 8:2. The batch size (batchsize) can be set according to needs, such as setting it to 64.

[0054] Figure 2b This is a flowchart of another method for generating counterfactual samples provided by the first embodiment of the present invention. Figure 2bAs shown, based on the normal net, a mirror net with a symmetrical network structure as the normal net can be constructed. Specifically, in order to enable the mirror net to output counterfactual samples, the mirror net can be symmetrical about the center of the feat layer of the normal net, that is, the mirror net can not contain the network structure with the classification layer (cls layer). The normal net and the mirror net can together constitute the counterfactual network model (Counter Factual Net, CF Net). After the CF Net is formed, the parameters of the model can be adjusted through the training sample set. There are many specific training methods. For example, the normal net and the mirror net can be trained separately; or, the normal net and the mirror net can be trained together; or, the mirror net learning can be completed through the normal net through the reinforcement learning framework, with the normal net as the learner backbone network and the mirror net as the discriminator backbone network.

[0055] Specifically, in an optional implementation of an embodiment of the present invention, based on the original machine learning network model, a mirror network model symmetrical to the original machine learning network model is constructed to generate a counterfactual network model, including: making all network layers between the input layer and the feature vector output layer in the original machine learning network model symmetrical about the center of the feature vector output layer to generate a network structure of the mirror network model; training the network structure of the original machine learning network model and the network structure of the mirror network model through a training sample set to generate a counterfactual network model.

[0056] in, Figure 2c This is a flowchart of generating counterfactual samples based on the PIM model provided by the first embodiment of the present invention. Figure 2cAs shown in the figure, when the PIM model uses AlexNet as the backbone, the feature extraction network (PIM_feat) of the PIM model has a total of 8 layers of networks. Among them, there are 3 types of network block structures (block): type 1block, type 2block and type 3block. Type 1block can be composed of a convolution layer, an activation layer and a pooling layer; type 2block can be composed of a convolution layer and an activation layer; type 3block can be composed of a fully connected layer. In an embodiment of the present invention, in order to improve the reliability of sample generation, the pooling layer can adopt maximum pooling (max_pooling), and the activation function uses a linear rectifier function (ReLU). Among them, max_pooling can be understood as the entire image being divided into several non-overlapping small blocks of the same size; in each small block, only the largest number is taken, and then after discarding other nodes, the original planar structure is maintained to obtain the output. The classification network (PIMCls) of the PIM model consists of three fully connected layers.

[0057] like Figure 2cAs shown, the PIM_feat mirror network model corresponding to the PIM model can be PIM_GAN. PIM_GAN also has 8 layers of blocks. The mirror layer corresponding to the type 1block in PIM_feat is f_type1block, the network structure is the same, and deconvolution and depooling operations are performed, but the input and output directions are opposite. For example, the number of input and output channels of the normal network layer is [inNums, outNums], while the number of input and output channels of the corresponding mirror network layer is [outNums, inNums]. Similarly, the mirror layer corresponding to the type 2block in PIM_feat is f_type 2block, which performs deconvolution operations; the mirror layer corresponding to the type 3block in PIM_feat is f_type 3block, which performs full connection operations. In an embodiment of the present invention, the method of constructing PIM_GAN can be to restore the maximum value during the depooling process and fill other positions with 0. The deconvolution layer adds padding as needed so that the output feature vector has the same dimension as the feature vector input by the symmetric network layer. Specifically, the order of the mirror network layers is symmetrical with the original network layers. The input is the feature vector generated by PIM_feat. After passing through three layers of f_type3block, the fully connected feature vector is input into a deconvolutional network consisting of one layer of f_type1block, two layers of f_type2block, and two layers of f_type1block. The output is the corresponding counterfactual sample. This setup creates a mirror network model, such as PIM_GAN, that is symmetrical about the feature vector output layer with respect to the original machine learning network model, such as the PIM model. The fully connected layers of PIM_feat and PIM_GAN are connected to form a counterfactual network model (PIM_CF). The PIM_CF model takes as input the training sample and outputs the corresponding counterfactual sample.

[0058] In an embodiment of the present invention, the pre-assembled PIM_CF model can be trained using a training sample set to adjust model parameters and generate a PIM_CF model. The training method can be various, such as the aforementioned separate training, joint training, or training through a reinforcement learning framework.

[0059] Step 220: Perform model training and testing on the original machine learning network model using the test sample set, and obtain the test results.

[0060] Among them, such as Figure 2bAs shown in the figure, the original machine learning network model includes a cls layer. Samples from the test sample set can be input into the trained CF Net, and the test results are obtained through the cls layer of the normal net in the CF Net. The test results can include samples that are correctly classified and samples that are incorrectly classified.

[0061] For example, samples in the test sample set can be pre-labeled with category labels. The computer can compare the category output by clslayer with the corresponding category label. If the comparison result shows that the category and category label are consistent, the corresponding sample test result can be considered a correctly classified sample; otherwise, the corresponding sample test result can be considered an incorrectly classified sample.

[0062] Specifically, such as Figure 2c As shown, the PIM model test sample set (PIM_testData) can be used to test the PIM model composed of PIM_feat and PIMCls in the PIM_CF model. For example, a test is performed after every 10,000 training iterations. Under the test macro rule (Macro), the test results are a precision of 0.798, a recall of 0.813, and a macro F1 value of 0.781. The number of TP samples (correctly identified as positive samples, which are actually positive samples) and TN samples (correctly identified as negative samples, which are actually negative samples) can be denoted as S1_T, S2_T, ... S1425_T, totaling 1425. The number of FP samples (incorrectly identified as positive samples, which are actually negative samples) and FN samples (incorrectly identified as negative samples, which are actually positive samples) can be denoted as F_S1426, F_S1427, ... F_S2000, totaling 575.

[0063] Step 230: Use the target sample that is incorrectly identified in the test result as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model.

[0064] like Figure 2b As shown in Figure 1, the target samples that are incorrectly identified in the test results can be input into CF Net to obtain the counterfactual samples output by CF Net.

[0065] Specifically, in an optional implementation of an embodiment of the present invention, the target sample incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model, including: using the target sample incorrectly identified in the test result as the input of the original machine learning network model in the counterfactual network model to obtain a target feature vector corresponding to the target sample; inputting the target feature vector into the mirror network model in the counterfactual network model to obtain the counterfactual sample output by the mirror network model.

[0066] For example, Figure 2c As shown, the FP samples and FN samples can be used as inputs to the PIM_feat model to obtain the feature vectors F_feat1426, F_feat1427…F_feat2000 output by PIM_feat; then F_feat1426, F_feat1427…F_feat2000 are input into the PIM_GAN model to obtain the corresponding counterfactual samples cf1, cf2…cf575.

[0067] On the basis of the above implementation mode, Figure 2b or Figure 2c As shown, the obtained counterfactual samples can be added to the training sample set, and the counterfactual network model can be repeatedly trained until the original machine learning network model in the counterfactual network model meets the preset conditions, such as the recognition accuracy exceeds the preset accuracy value. The technical solution of the embodiment of the present invention can be used to expand the small samples in the prior art, which is conducive to improving the recognition accuracy of the machine learning model.

[0068] The technical solution of the embodiment of the present invention generates a counterfactual network model by constructing a mirror network model symmetrical to the original machine learning network model based on the original machine learning network model; performing model training and testing on the original machine learning network model through a test sample set to obtain test results; using the target samples that were incorrectly identified in the test results as the input of the counterfactual network model to obtain counterfactual samples output by the counterfactual network model, thereby solving the problem of poor recognition rate of the machine learning model due to small samples in the training of the machine learning model, and realizing the expansion of small samples that does not rely on human experience, that is, generating counterfactual samples, which is beneficial to improving the accuracy and robustness of the machine learning model from the source of machine learning.

[0069] Example 2

[0070] Figure 3 This is a flowchart of a counterfactual sample generation method provided by Example 2 of the present invention. The implementation method in this embodiment may be a refinement of the technical solution of the above embodiment, and may be combined with one or more of the above implementation methods, such as Figure 3 As shown, the method specifically includes:

[0071] Step 310: All network layers between the input layer and the feature vector output layer in the original machine learning network model are symmetrical about the feature vector output layer to generate a network structure of a mirror network model.

[0072] Step 320: Train the network structure of the original machine learning network model and the network structure of the mirror network model using the training sample set to generate a counterfactual network model.

[0073] In an optional implementation of an embodiment of the present invention, the network structure of the original machine learning network model and the network structure of the mirror network model are trained through a training sample set to generate a counterfactual network model, including: pre-training the network structure of the original machine learning network model through the training sample set to generate the original machine learning network model; inputting samples in the training sample set into the trained original machine learning network model to obtain a feature vector output by a feature vector output layer of the original machine learning network model; using the feature vector as input when training the network structure of the mirror network model, performing mirror network model training, and generating a counterfactual network model.

[0074] In the embodiment of the present invention, the counterfactual network model can be generated by separately training the normal net and the mirror net. Specifically, the normal net can be trained with a training sample set first, and then the mirror net can be trained based on the pre-trained normal net. Figure 2b As shown, the normal net can have a loss function (cls loss), and the normal net can be repeatedly iterated forward and backward according to the training sample set to update the model parameters until the cls loss converges to generate a pre-trained normal net. Figure 2b As shown, the mirror net can have a loss function, and its loss function can represent the difference between the counterfactual sample and the sample in the training sample set. The loss function of the mirror net can adopt the MES loss used by networks such as U-Net, AutoEncoder or GAN. Figure 2b As shown in the figure, the samples in the training sample set can be input into the pre-trained normal net, and the corresponding feature vectors are obtained from the feat layer of the pre-trained normal net and input into the mirror net. Through repeated iterations of forward and backward calculations, the mirror net model parameters are updated until the MES loss converges, and the mirror net is generated, thereby completing the learning of the CF Net.

[0075] For example, Figure 2cAs shown in the figure, when learning PIM_CF based on the pre-trained PIM_feat model, the PIM_feat training sample set is used for PIM_CF learning. For example, a minibatch of samples (S1, S2…S64) can be constructed with a batch size of 64. After the PIM_feat forward calculation, the feature vectors feat1, feat2…feat64 are obtained. The feature vectors feat1, feat2…feat64 are used as the input of the PIM_GAN. The PIM_GAN forward calculation outputs the counterfactual samples S1_CF, S2_CF…S64_CF. The MSE loss is calculated between the counterfactual samples S1_CF, S2_CF…S64_CF and the corresponding input samples S1, S2…S64. The reverse calculation is performed based on the MSE loss to update the parameters of the PIM_GAN network. This process is repeated until the MSE loss reaches the convergence condition. After the training of this step is completed, the PIM_GAN model can be obtained. By inputting the feature vector output by the PIM_feat model, the counterfactual sample corresponding to the input sample of the PIM_feat model can be obtained.

[0076] Based on the above implementation, when the pre-trained normal net undergoes a model update, the mirror net model also needs to be updated synchronously. To synchronize and automate model updates, an iterative learning framework or adversarial learning framework can be introduced between the normal net and the mirror net to enable automated mirror net updates.

[0077] In an optional implementation of an embodiment of the present invention, the network structure of the original machine learning network model and the network structure of the mirror network model are trained through a training sample set to generate a counterfactual network model, including: using a training sample set, using a multi-task learning method to simultaneously train the network structure of the original machine learning network model and the network structure of the mirror network model to generate a counterfactual network model.

[0078] Among them, in the embodiment of the present invention, CF Net can be generated by training normal net and mirror net at the same time. Specifically, normal net and mirror net can be trained at the same time through multi-task learning. For example, a combination of AutoEncoder (self-encoder), VAE (variational encoder), U-Net (U-shaped structure network), GAN (generative adversarial network) and the like can be used to train mirror net. Figure 2bAs shown in the figure, normal net can have cls loss and mirror net can have MSE loss. The model parameters of normal net and mirror net can be updated through repeated iterations of forward and backward calculations until both cls loss and MSE loss converge to generate CF Net.

[0079] Step 330: Perform model training and testing on the original machine learning network model using the test sample set, and obtain the test results.

[0080] Step 340: Use the target sample that is incorrectly identified in the test result as the input of the original machine learning network model in the counterfactual network model to obtain a target feature vector corresponding to the target sample.

[0081] Step 350: Input the target feature vector into the mirror network model in the counterfactual network model to obtain the counterfactual sample output by the mirror network model.

[0082] In an optional implementation of an embodiment of the present invention, before inputting the target feature vector into the mirror network model in the counterfactual network model to obtain the counterfactual sample output by the mirror network model, it also includes: adding a disturbance value to the target feature vector to expand the target feature vector.

[0083] The perturbation value can be a fine-tuning of the target feature vector. For example, the feature vector can be used as the Z factor of the GAN network. Alternatively, the perturbation value can be a slight increase or decrease in one or more values ​​in the target feature vector. After fine-tuning the feature vector, it can be fed into the mirror net to generate more counterfactual samples.

[0084] The technical solution of the embodiment of the present invention generates a network structure of a mirror network model by making all network layers between the input layer and the feature vector output layer in the original machine learning network model symmetrical about the center of the feature vector output layer; trains the network structure of the original machine learning network model and the network structure of the mirror network model through a training sample set to generate a counterfactual network model; performs model training and testing on the original machine learning network model through a test sample set to obtain a test result; uses the target sample that is incorrectly identified in the test result as the input of the original machine learning network model in the counterfactual network model to obtain a target feature vector corresponding to the target sample; inputs the target feature vector into the mirror network model in the counterfactual network model to obtain a counterfactual sample output by the mirror network model, which solves the problem of poor recognition rate of the machine learning model due to small samples in the training of the machine learning model, can integrate the experience feedback mechanism into the traditional machine learning process, and innovatively proposes a general and automated method to enrich small samples by generating counterfactual samples in a targeted manner, thereby facilitating the optimization of the learning stage.

[0085] Example 3

[0086] Figure 4 This is a flowchart of a counterfactual network model adjustment method provided in Example 3 of the present invention. This embodiment can be applied to the situation where small samples are expanded in machine learning model training, counterfactual samples are generated, and the machine learning model is optimized and adjusted to improve the model recognition accuracy and robustness. The method can be executed by a counterfactual network model adjustment device, which can be implemented by software and / or hardware, and the device can be integrated in a computer.

[0087] like Figure 4 As shown, the method specifically includes:

[0088] Step 410: Generate a counterfactual sample using the counterfactual sample generation method provided by any embodiment of the present invention.

[0089] Step 420: Add the counterfactual samples to the original training sample set to generate a target training sample set, and use the target training sample set to adjust the counterfactual network model provided by any embodiment of the present invention.

[0090] The target training sample set includes the original samples and the generated counterfactual samples.

[0091] Step 430: Perform model training and testing on the original machine learning network model in the counterfactual network model using the test sample set to obtain a test result.

[0092] The test sample set may be the original test sample, or some counterfactual samples may be added, which is not specifically limited in the embodiment of the present invention.

[0093] Step 440: If the test result does not meet the preset model training conditions, the target sample that is incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model; and the operation of adding the counterfactual sample to the original training sample set to generate the target training sample set is returned until the test result meets the preset model training conditions and the model adjustment is stopped.

[0094] The preset model training condition may be that the accuracy of sample type recognition is greater than or equal to a preset accuracy rate. When the recognition accuracy rate is low, repeated counterfactual sample generation and repeated model adjustments may be performed to continuously improve the accuracy and robustness of the model.

[0095] The technical solution of the embodiment of the present invention generates counterfactual samples by adopting the counterfactual sample generation method provided by any embodiment of the present invention; adds the counterfactual samples to the original training sample set to generate a target training sample set, and uses the target training sample set to adjust the counterfactual network model provided by any embodiment of the present invention; performs model training and testing on the original machine learning network model in the counterfactual network model through the test sample set to obtain a test result; performs model training and testing on the original machine learning network model in the counterfactual network model through the test sample set to obtain a test result, which solves the problem of poor recognition rate of machine learning models caused by small samples in machine learning model training, can integrate the experience feedback mechanism into the traditional machine learning process, and innovatively proposes a general and automated method, which optimizes the training set by generating counterfactual samples in a targeted manner, and feeds back the experience of the testing phase to the learning phase, thereby reducing a large amount of manual work and improving the engineering application efficiency of machine learning, so that the shortcomings of the current machine learning model can be automatically and targetedly improved and adjusted, thereby improving the model accuracy and enhancing the robustness of the model.

[0096] Example 4

[0097] Figure 5 Schematic diagram of the structure of a counterfactual sample generation device provided by the fourth embodiment of the present invention. In conjunction with the figure, the device includes: a counterfactual network model generation module 510, a test result acquisition module 520 and a counterfactual sample acquisition module 530. Among them:

[0098] A counterfactual network model generation module 510 is configured to construct a mirror network model symmetrical to the original machine learning network model based on the original machine learning network model to generate a counterfactual network model;

[0099] The test result acquisition module 520 is used to perform model training test on the original machine learning network model through the test sample set and obtain the test results;

[0100] The counterfactual sample acquisition module 530 is used to use the target sample that is incorrectly identified in the test result as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model.

[0101] Optionally, the counterfactual network model generation module 510 includes:

[0102] A mirror network model generation unit is used to make all network layers between the input layer and the feature vector output layer in the original machine learning network model symmetrical about the feature vector output layer to generate a network structure of the mirror network model;

[0103] The counterfactual network model generation unit is used to train the network structure of the original machine learning network model and the network structure of the mirror network model through a training sample set to generate a counterfactual network model.

[0104] Optional, counterfactual network model generation unit, including:

[0105] The original machine learning network model generation subunit is used to pre-train the network structure of the original machine learning network model through the training sample set to generate the original machine learning network model;

[0106] The feature vector acquisition subunit is used to input the samples in the training sample set into the trained original machine learning network model to obtain the feature vector output by the feature vector output layer of the original machine learning network model;

[0107] The counterfactual network model generation subunit is used to use the feature vector as input when training the network structure of the mirror network model, perform mirror network model training, and generate a counterfactual network model.

[0108] Optional, counterfactual network model generation unit, including:

[0109] Another counterfactual network model generation subunit is used to generate a counterfactual network model by simultaneously training the network structure of the original machine learning network model and the network structure of the mirror network model using a multi-task learning method through a training sample set.

[0110] Optionally, the counterfactual sample acquisition module 530 includes:

[0111] a target feature vector acquisition unit, configured to use the target sample that was incorrectly identified in the test result as the input of the original machine learning network model in the counterfactual network model to obtain a target feature vector corresponding to the target sample;

[0112] The counterfactual sample acquisition unit is used to input the target feature vector into the mirror network model in the counterfactual network model to obtain the counterfactual sample output by the mirror network model.

[0113] Optionally, the device further includes:

[0114] The perturbation value adding module is used to add a perturbation value to the target feature vector before inputting the target feature vector into the mirror network model in the counterfactual network model to obtain the counterfactual sample output by the mirror network model, so as to expand the target feature vector.

[0115] The counterfactual sample generation device provided in the embodiment of the present invention can execute the counterfactual sample generation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0116] Example 5

[0117] Figure 6 This is a schematic diagram of the structure of a counterfactual network model adjustment device provided by the fifth embodiment of the present invention. Figure 6 The device includes: a counterfactual sample generation module 610, a counterfactual network model adjustment module 620, a test result acquisition module 630 and a model adjustment stop module 640.

[0118] A counterfactual sample generation module 610, configured to generate counterfactual samples using the counterfactual sample generation method provided by any embodiment of the present invention;

[0119] A counterfactual network model adjustment module 620 is configured to add counterfactual samples to the original training sample set to generate a target training sample set, and to use the target training sample set to adjust the counterfactual network model provided by any embodiment of the present invention;

[0120] A test result acquisition module 630 is used to perform model training and testing on the original machine learning network model in the counterfactual network model using a test sample set to obtain a test result;

[0121] The model adjustment stop module 640 is used to use the target sample that was incorrectly identified in the test result as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model if the test result does not meet the preset model training conditions; and return to the operation of adding the counterfactual sample to the original training sample set to generate the target training sample set until the test result meets the preset model training conditions and stops model adjustment.

[0122] The counterfactual network model adjustment device provided in the embodiment of the present invention can execute the counterfactual network model adjustment method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0123] Example 6

[0124] Figure 7This is a schematic diagram of the structure of a computer device provided by Example 6 of the present invention. Figure 7 As shown, the device includes:

[0125] one or more processors 710, Figure 7 A processor 710 is taken as an example;

[0126] Memory 720;

[0127] The device may further include an input device 730 and an output device 740 .

[0128] The processor 710, memory 720, input device 730 and output device 740 in the device can be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0129] The memory 720 is a non-transitory computer-readable storage medium that can be used to store software programs, computer executable programs, and modules, such as a method for generating counterfactual samples in an embodiment of the present invention, or a program instruction / module corresponding to a method for adjusting a counterfactual network model (e.g., the attached Figure 5 The counterfactual network model generation module 510, the test result acquisition module 520 and the counterfactual sample acquisition module 530 shown in FIG; or Figure 6 The processor 710 executes the software programs, instructions, and modules stored in the memory 720 to execute various functional applications and data processing of the computer device, thereby implementing a counterfactual sample generation method of the above-mentioned method embodiment, namely:

[0130] Based on the original machine learning network model, construct a mirror network model that is symmetrical to the original machine learning network model to generate a counterfactual network model;

[0131] Performing model training and testing on the original machine learning network model using a test sample set, and obtaining test results;

[0132] The target sample that is incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model.

[0133] Alternatively, a counterfactual network model adjustment approach is:

[0134] Generate a counterfactual sample using the counterfactual sample generation method provided by any embodiment of the present invention;

[0135] Adding the counterfactual sample to the original training sample set to generate a target training sample set, and using the target training sample set to adjust the counterfactual network model provided by any embodiment of the present invention;

[0136] Performing model training and testing on the original machine learning network model in the counterfactual network model using a test sample set to obtain a test result;

[0137] If the test result does not meet the preset model training conditions, the target sample that is incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model; and the operation of adding the counterfactual sample to the original training sample set to generate the target training sample set is returned until the test result meets the preset model training conditions and the model adjustment is stopped.

[0138] The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 720 may optionally include a memory remotely located relative to the processor 710, and these remote memories may be connected to the terminal device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0139] The input device 730 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the computer device. The output device 740 may include a display device such as a display screen.

[0140] Example 7

[0141] Embodiment 7 of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, a counterfactual sample generation method as provided in an embodiment of the present invention is implemented, namely:

[0142] Based on the original machine learning network model, construct a mirror network model that is symmetrical to the original machine learning network model to generate a counterfactual network model;

[0143] Performing model training and testing on the original machine learning network model using a test sample set, and obtaining test results;

[0144] The target sample that is incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model.

[0145] Alternatively, a counterfactual network model adjustment approach is:

[0146] Generate a counterfactual sample using the counterfactual sample generation method provided by any embodiment of the present invention;

[0147] Adding the counterfactual sample to the original training sample set to generate a target training sample set, and using the target training sample set to adjust the counterfactual network model provided by any embodiment of the present invention;

[0148] Performing model training and testing on the original machine learning network model in the counterfactual network model using a test sample set to obtain a test result;

[0149] If the test result does not meet the preset model training conditions, the target sample that is incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model; and the operation of adding the counterfactual sample to the original training sample set to generate the target training sample set is returned until the test result meets the preset model training conditions and the model adjustment is stopped.

[0150] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0151] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0152] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0153] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0154] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for generating counterfactual samples, characterized in that: include: According to the original machine learning network model, a mirror network model symmetrical to the original machine learning network model is constructed to generate a counterfactual network model; wherein the original machine learning network model is a plant species identification model PIM, the input of the original machine learning network model is a plant photo, and the output is the category to which the plant belongs; the feature extraction network PIM_feat of the PIM model includes three network block structures: type 1block, type 2block and type 3block, type 1block consists of a convolutional layer, an activation layer and a pooling layer, type 2block consists of a convolutional layer and an activation layer, and type 3block consists of a fully connected layer; the mirror network model PIM_GAN of PIM includes f_type1block, f_type2block and f_type3block, f_type1block is a mirror layer of type 1block in PIM_feat, has the same network structure, performs deconvolution and depooling operations, but has opposite input and output directions, f_type2block is a mirror layer of type 2block in PIM_feat, performs deconvolution operations, and f_type3block is type The mirror layer of 3block is fully connected; the arrangement order of each mirror layer is symmetrical with the original network layer; the fully connected layer of PIM_feat and the fully connected layer of PIM_GAN are connected to form the counterfactual network model PIM_CF; Performing model training and testing on the original machine learning network model using a test sample set, and obtaining test results; wherein the test sample set is composed of collected plant photos and species labels; The target sample that is incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model.

2. The method according to claim 1, characterized in that Based on the original machine learning network model, a mirror network model symmetrical to the original machine learning network model is constructed to generate a counterfactual network model, including: The network structure of the original machine learning network model and the network structure of the mirror network model are trained using a training sample set to generate a counterfactual network model.

3. The method according to claim 2, characterized in that The network structure of the original machine learning network model and the network structure of the mirror network model are trained using a training sample set to generate a counterfactual network model, including: Pre-training the network structure of the original machine learning network model through a training sample set to generate the original machine learning network model; Inputting samples in the training sample set into the trained original machine learning network model to obtain a feature vector output by a feature vector output layer of the original machine learning network model; The feature vector is used as input for network structure training of the mirror network model, and the mirror network model is trained to generate a counterfactual network model.

4. The method according to claim 2, characterized in that The network structure of the original machine learning network model and the network structure of the mirror network model are trained using a training sample set to generate a counterfactual network model, including: By using a training sample set, a multi-task learning method is adopted to simultaneously train the network structure of the original machine learning network model and the network structure of the mirror network model to generate a counterfactual network model.

5. The method according to claim 1, wherein Using the target sample that is incorrectly identified in the test result as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model includes: Using the target sample that is incorrectly identified in the test result as the input of the original machine learning network model in the counterfactual network model to obtain a target feature vector corresponding to the target sample; The target feature vector is input into the mirror network model in the counterfactual network model to obtain the counterfactual sample output by the mirror network model.

6. The method according to claim 5, characterized in that Before inputting the target feature vector into the mirror network model in the counterfactual network model to obtain the counterfactual sample output by the mirror network model, the method further includes: A disturbance value is added to the target feature vector to expand the target feature vector.

7. A method for adjusting a counterfactual network model, characterized in that: include: Generating a counterfactual sample using the counterfactual sample generation method according to any one of claims 1 to 6; Adding the counterfactual sample to an original training sample set to generate a target training sample set, and using the target training sample set to adjust the counterfactual network model according to any one of claims 1 to 6; Performing model training and testing on the original machine learning network model in the counterfactual network model using a test sample set to obtain a test result; If the test result does not meet the preset model training conditions, the target sample that is incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model; And return to the operation of adding the counterfactual sample to the original training sample set to generate the target training sample set, until the test result meets the preset model training conditions, and stop model adjustment; Among them, the original machine learning network model is a plant species recognition model, the input of the original machine learning network model is a plant photo, and the output is the category to which the plant belongs; the training sample set and the test sample set are both composed of collected plant photos and species labels.

8. A computer device comprising a processor and a memory, wherein the memory is configured to store instructions that, when executed, cause the processor to perform the following operations: According to the original machine learning network model, a mirror network model symmetrical to the original machine learning network model is constructed to generate a counterfactual network model; wherein, The original machine learning network model is a plant species identification model PIM, the input of the original machine learning network model is a plant photo, and the output is the category to which the plant belongs; the feature extraction network PIM_feat of the PIM model includes three network block structures: type 1block, type 2block and type 3block, type 1block consists of a convolution layer, an activation layer and a pooling layer, type 2block consists of a convolution layer and an activation layer, and type 3block consists of a fully connected layer; the mirror network model PIM_GAN of PIM includes f_type1block, f_type2block and f_type3block, f_type1block is a mirror layer of type 1block in PIM_feat, has the same network structure, performs deconvolution and depooling operations, but has opposite input and output directions, f_type2block is a mirror layer of type 2block in PIM_feat, performs deconvolution operations, and f_type3block is type The mirror layer of 3block is fully connected; the arrangement order of each mirror layer is symmetrical with the original network layer; the fully connected layer of PIM_feat and the fully connected layer of PIM_GAN are connected to form the counterfactual network model PIM_CF; Performing model training and testing on the original machine learning network model using a test sample set, and obtaining test results; wherein the test sample set is composed of collected plant photos and species labels; The target sample that is incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model.

9. The computer device according to claim 8, wherein: The processor is configured to generate a counterfactual network model by: The network structure of the original machine learning network model and the network structure of the mirror network model are trained using a training sample set to generate a counterfactual network model.

10. The computer device according to claim 9, wherein: The processor is configured to generate a counterfactual network model by: Pre-training the network structure of the original machine learning network model through a training sample set to generate the original machine learning network model; Inputting samples in the training sample set into the trained original machine learning network model to obtain a feature vector output by a feature vector output layer of the original machine learning network model; The feature vector is used as input for network structure training of the mirror network model, and the mirror network model is trained to generate a counterfactual network model.

11. The computer device according to claim 9, wherein: The processor is configured to generate a counterfactual network model by: By using a training sample set, a multi-task learning method is adopted to simultaneously train the network structure of the original machine learning network model and the network structure of the mirror network model to generate a counterfactual network model.

12. The computer device according to claim 8, wherein: The processor is configured to obtain counterfactual samples by: Using the target sample that is incorrectly identified in the test result as the input of the original machine learning network model in the counterfactual network model to obtain a target feature vector corresponding to the target sample; The target feature vector is input into the mirror network model in the counterfactual network model to obtain the counterfactual sample output by the mirror network model.

13. The computer device according to claim 12, wherein: The processor is configured to expand the counterfactual sample before inputting the target feature vector into the mirror network model in the counterfactual network model to obtain the counterfactual sample output by the mirror network model in the following manner: A disturbance value is added to the target feature vector to expand the target feature vector.

14. A computer device comprising a processor and a memory, wherein the memory is configured to store instructions that, when executed, cause the processor to: Generating a counterfactual sample using the counterfactual sample generation method according to any one of claims 1 to 6; Adding the counterfactual sample to an original training sample set to generate a target training sample set, and using the target training sample set to adjust the counterfactual network model according to any one of claims 1 to 6; Performing model training and testing on the original machine learning network model in the counterfactual network model using a test sample set to obtain a test result; If the test result does not meet the preset model training conditions, the target sample that is incorrectly identified in the test result is used as the input of the counterfactual network model to obtain the counterfactual sample output by the counterfactual network model; And return to the operation of adding the counterfactual sample to the original training sample set to generate the target training sample set, until the test result meets the preset model training conditions, and stop model adjustment; Among them, the original machine learning network model is a plant species recognition model, the input of the original machine learning network model is a plant photo, and the output is the category to which the plant belongs; the training sample set and the test sample set are both composed of collected plant photos and species labels.

15. A storage medium for storing instructions, wherein the instructions are used to execute the counterfactual sample generation method according to any one of claims 1 to 6; or, the instructions are used to execute the counterfactual network model adjustment method according to claim 7.

Images