A self-developing neural network design method and storage medium inspired by DNA damage repair mechanisms

Abstract

This invention discloses a self-developing neural network design method and storage medium inspired by DNA damage repair mechanisms. The method includes selecting a neural network model and defining its hyperparameters; acquiring an image dataset and dividing it into training, validation, and test sets to form a sequence of tasks; training the neural network model based on the first task data in the training set; testing the trained neural network model based on the first task data in the validation set; performing self-developing growth on the neural network model in both width and depth directions to increase its size; repeating the training and testing process and determining whether further increasing the neural network model size is necessary; if growth stops, obtaining the model's classification accuracy based on the task data in the test set; repeating the process until all task data in the test set has been tested. This invention promotes the intelligent development of neural networks in image classification tasks and their practical applications by dynamically adjusting the size of the neural network model and growing new neurons during training to cope with constantly changing tasks and environments.

Description

Technical Field

[0001] This invention relates to the field of neural network image processing technology, and in particular to a design method and storage medium for a self-developing neural network inspired by DNA damage repair mechanisms. Background Technology

[0002] In traditional neural network image classification models, the model structure is determined from the outset and cannot be changed during training and testing, which limits its practical application. In reality, data and tasks are often presented sequentially over time. Simple image classification tasks require only small-scale model structures, while complex tasks require larger-scale models. To improve model utilization and maximize performance, different network sizes should be used for different image classification tasks.

[0003] Secondly, when processing sequential tasks, neural network image classification models not only exhibit forgetting of old tasks but also a lack of learning ability, meaning their plasticity decreases. The same model achieves excellent performance on non-sequential tasks, but its overall performance drops significantly across all data when handling sequential tasks, limiting the adaptability of neural network image classification models to new tasks and environments. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a self-developing neural network design method and storage medium inspired by DNA damage repair mechanisms. This method involves dynamically adjusting the scale of the neural network model during training and growing new neurons to cope with constantly changing tasks and environments, thereby promoting the intelligent development of neural networks in image classification tasks and their practical applications.

[0005] Technical Solution: To achieve the above objectives, the present invention provides a self-developing neural network design method inspired by DNA damage repair mechanisms, comprising the following steps:

[0006] S1: Initialize the parameters and weights θ of the neural network model, and define the maximum number of times the neural network model can grow N, the performance improvement threshold ε, the model learning rate α, the number of training epochs, and the number of learning categories class_task for each task;

[0007] S2: Obtain an image dataset containing image classification labels, and based on the class_task value, divide the image dataset into a training set, a validation set, and a test set to form a sequence task;

[0008] S3: Train the neural network model based on the first task data in the training set and the learning rate α and the number of training epochs defined in S1;

[0009] S4: Based on the first task data in the validation set, test the neural network model trained in S3 and obtain the classification accuracy C of the model on the task data.

[0010] S5: Perform self-growth on the neural network model defined in S1 in both width and depth directions to increase the size of the neural network model;

[0011] S6: Repeat S3 and S4 to obtain the new model's classification accuracy C' on the task data. If C'-C>ε, repeat S3-S5 to continue increasing the size of the neural network model. If C'-C<ε and the number of times the neural network model grows in the depth direction is greater than the defined maximum number of growth times N, stop the growth of the neural network model on the current task and test it based on the first task data of the test set to obtain the model's classification accuracy T1 on the task data.

[0012] S7: Based on the second task data in the training set, repeat S3-S6 to obtain the classification accuracy T2 of the model on that task data;

[0013] S8: Repeat S7 until all image classification tasks have been tested, obtain all classification accuracies of the self-growth neural network model on the test set, and evaluate the performance of the self-growth neural network model based on the classification accuracy.

[0014] The neural network model described in S1 has an architecture of either a convolutional neural network (CNN) or a linear layer (Linear).

[0015] In S2, the sequence task partitioning is determined based on the total number of classes in the image dataset (class_num) and the number of classes learned for each task (class_task). Specifically, the training set, validation set, and test set are each divided into class_num / class_task tasks, with the number of classes for each task increasing sequentially. That is, the first task has class_task classes, the second task has 2*class_task classes, and the last task has class_num classes.

[0016] In S5, the neural network model undergoes self-growth, prioritizing growth in the width direction and then in the depth direction. Growth in the width direction involves adding new neurons or new convolutional kernel channels to the penultimate layer of the neural network model, while growth in the depth direction involves inserting new linear or convolutional layers into the penultimate layer of the neural network model.

[0017] For CNN architecture neural network models, width growth refers to increasing the number of channels in the convolutional kernel, while depth growth refers to increasing the number of convolutional layers in the entire neural network model. The number of channels in the newly added convolutional layer is set to half the number of channels in the convolutional kernel of the previous layer of the model. The last layer of the CNN architecture neural network model is the classification head, which has a fixed size and only learns weight values.

[0018] For a neural network model with a Linear architecture, width growth refers to increasing the number of neurons in the same layer, while depth growth refers to increasing the number of Linear layers in the entire neural network model. The number of neurons in the newly added Linear layer is set to half the number of neurons in the previous layer of the model.

[0019] The self-growth process of the neural network model consists of two steps: the first step is to grow a new network structure, and the second step is to merge the original network structure with the newly added network structure. For a CNN architecture neural network model, the parameter before growth is μ. 11 =[out ch ,in ch kernel size kernel size First, increase the number of new convolutional channels: μ 12 =[new ch ,in ch kernel size kernel size ], and then concatenate them to obtain the new weight μ1 = [out ch +new ch ,in ch kernel size kernel size ];

[0020] For the Linear architecture, the parameter before growth is μ. 21 =[out fea ,in fea First, increase the number of new neurons: μ 22 =[new fea ,in fea ,], then concatenate to obtain the new weight μ2 = [out fea +new fea ,in fea ].

[0021] The present invention provides a computer-readable storage medium for storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform the self-developing neural network design method inspired by the DNA damage repair mechanism described above.

[0022] Beneficial effects: The present invention has the following advantages: 1. The present invention is inspired by the DNA damage repair mechanism. By dynamically adjusting the scale of the neural network model during the training process, it achieves the effect of adaptively changing the model scale according to the task complexity. At the same time, by growing a new number of neurons to cope with the constantly changing tasks and environments, it promotes the accumulation and inheritance of knowledge by the neural network model, thereby improving the robustness and adaptability of the model in the image classification process, further reducing training costs and resource consumption, and promoting the intelligent development of neural networks in image classification tasks and their practical application.

[0023] 2. This invention utilizes smaller model parameters to generate complex neural networks, alleviating the lack of plasticity in deep learning models for sequence tasks and improving the model's adaptability and learning ability for new tasks. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the process of the present invention;

[0025] Figure 2 This is a flowchart of the growth process of a self-developing neural network.

[0026] Figure 3 The results of self-growth training based on a CNN architecture on the CIFAR100 dataset;

[0027] Figure 4 The training results for self-growth based on a CNN architecture on the CIFAR10 dataset;

[0028] Figure 5 This is a feature similarity matrix between different layers of a network grown using a Linear architecture for the CIFAR10 dataset. Detailed Implementation

[0029] The technical solution of the present invention will be described in detail below with reference to the embodiments and accompanying drawings.

[0030] like Figure 1 As shown, the self-developing neural network design method inspired by DNA damage repair mechanisms described in this invention includes the following:

[0031] S1: Select either CNN or Linear neural network architecture, initialize the model's parameter weights θ, define the maximum number of network growth iterations N, the performance improvement threshold ε, the model learning rate α, the number of training epochs, the number of learning classes per task class_task, and initialize the number of depth growth iterations num_depth to 0.

[0032] S2: Based on the `class_task` value in S1, the dataset is divided into a training set, a validation set, and a test set, forming a sequence of tasks. This is determined by the total number of classes in the dataset (`class_num`) and the number of classes learned for each task (`class_task`). Specifically, the training set, validation set, and test set are each divided into `class_num / class_task` tasks, with the number of classes in each task increasing sequentially. That is, the first task has `class_task` classes, the second task has 2 * `class_task` classes, and the last task has `class_num` classes.

[0033] S3: Based on the first task data in the training set in S2, the network model defined in S1 is trained using the learning rate α and the number of training epochs in S1. The network is trained epoch times for each growth.

[0034] S4: The model trained using S3 is tested on the first task of the validation set in S2, and its classification accuracy is C.

[0035] S5: Perform self-growth growth on the network defined in S1 in both width and depth directions to increase the model size. The specific growth method is as follows: prioritize growth in the width direction, and then grow in the depth direction. For CNN architecture, width growth means increasing the number of channels in the convolutional kernel, with the number of channels increased by 5. Depth growth means increasing the number of convolutional layers in the network, with the number of channels in the new convolutional layer being half the number of channels in the previous layer. For Linear architecture, width growth means increasing the number of neurons in the same layer, with the number of neurons increased by 50. Depth growth means increasing the number of Linear layers in the network, with the number of neurons in the new Linear layer being half the number of neurons in the previous layer.

[0036] Growing in the width direction involves adding new neurons or new convolutional kernel channels to the penultimate layer of the model. Growing in the depth direction involves inserting new linear or convolutional layers into the penultimate layer of the model. The last layer of the model is the classification head, which has a fixed size and only learns its weight values.

[0037] The self-growth process of a neural network model consists of two steps: the first step is to grow a new network structure, and the second step is to merge the original network structure with the newly added network structure. For a CNN architecture, the parameter before growth is μ1 = [out]. ch ,in ch kernel size kernel size First, increase the number of convolution channels, μ2 = [new ch ,in ch kernelsize kernel size Then concatenate them to obtain the new weight μ = [out] ch +new ch ,in ch kernel size kernel size For the Linear architecture, the parameters before growth are μ1 = [out] fea ,in fea First, the number of new neurons increases, μ2 = [new] fea ,in fea Then concatenate the ,] to obtain the new weight μ = [out] fea +new fea ,in fea ].

[0038] S6: Repeat steps S3 and S4 to obtain a new classification accuracy of C' for this task; if C'-C>ε, repeat steps S3-S5 to continue increasing the network size; if C'-C<ε and the depthwise growth count num_depth is incremented by one, and if num_depth is greater than N defined in S1, stop the model growth on the current task and test it on the first task of the test set in S2 to obtain a classification accuracy of T1; Figure 2 The diagram shown illustrates the growth process of a self-developing neural network.

[0039] S7: Based on the second task data in the training set in S2, repeat steps S3-S6 to obtain its classification accuracy as T2.

[0040] S8: Repeat step S7 until all tasks have been tested and the classification accuracy T1, T2...TN of the test set is obtained.

[0041] To evaluate the rationality of the final grown network structure, after the model has learned all the data, similarity calculations are performed on the features learned between different layers of the model, and on the features between different neurons within the same layer. The calculation method is as follows:

[0042]

[0043] Where, x i ,x j H represents the features of different layers of a network or the features of different neurons in the same layer of a network. H is a normalized matrix, and its calculation method is as follows:

[0044]

[0045] I n Let x be an identity matrix of dimension n. iThe specific calculation method is as follows: based on the model θ after training all tasks, 1000 samples are randomly selected from the test set, and forward propagation is performed to obtain the features of different network layers. For the Linear layer, the feature dimension is (batch, fea_dim), where batch is the batch size and fea_dim is the feature, which is directly calculated using the CKA function. For the CNN layer, the obtained feature dimension is (batch, out_ch, height, width), which needs to be averaged to obtain a two-dimensional dimension, that is, the average is taken on the weight and width dimensions to obtain the (batch, out_ch) features, and then the similarity is calculated using the CKA function.

[0046] Taking the CIFAR100 and CIFAR10 datasets as examples, the self-growth training results of the CNN architecture based on the CIFAR100 dataset are as follows: Figure 3 As shown, based on the CIFAR10 dataset, the self-growth training results of the CNN architecture are as follows: Figure 4 As shown.

[0047] The CIFAR10 dataset contains 10 categories, with 2 categories per task, for a total of 5 tasks. The CIFAR100 dataset contains 100 categories, with 5 categories per task, for a total of 20 tasks. The seq S-DNN dataset represents the test results of the self-developing neural network on sequence tasks, the seq baseline dataset represents the test results of the traditional neural network on sequence tasks, the baseline S-DNN dataset represents the test results of the self-developing neural network on non-sequence tasks, and the baseline dataset represents the test results of the traditional neural network on non-sequence tasks. Experimental results show that the self-developing neural network alleviates the problem of lack of plasticity in neural networks when facing sequence tasks. That is, it is not that the model's expressive ability is insufficient, but rather that its learning ability declines when learning sequence tasks. Both datasets show a decline in test performance when learning sequence tasks, but the decline in performance of the self-developing neural network is slower, mitigating the drawbacks of lack of plasticity.

[0048] Figure 5 A self-developing network was grown on the CIFAR10 dataset using a linear architecture. The feature similarity matrix between different layers was calculated based on CKA. The results show that the similarity of the learned features between different layers is very low, which verifies the rationality of the self-developing neural network.

Claims

1. A DNA damage repair mechanism inspired self-developing neural network design method, characterized in that, The method comprises the following steps: S1: initialize the parameter weight of the neural network model , define the maximum growth number N of the neural network model, the performance improvement threshold , the model learning rate , the model training round number Epoch, and the learning class number of each task ; S2: Obtain an image dataset containing image classification labels, and divide the image dataset into a training set, a validation set, and a test set based on values to form a sequence task; S3: based on the first task data in the training set and the learning rate defined in S1 , the number of model training rounds Epoch, to train the neural network model; S4: based on the first task data in the verification set, test the neural network model trained in S3 to obtain the classification accuracy of the model on the task data ; S5: performing self-growth of the neural network model defined in S1 in the width and depth directions to increase the size of the neural network model; the neural network model is grown by self-growth, and is grown in the width direction first and then in the depth direction; the growth in the width direction is to increase the number of neurons or the number of new convolution kernel channels in the penultimate layer of the neural network model, and the growth in the depth direction is to insert a new Linear layer or a convolution layer in the penultimate layer of the neural network model; S6: repeat S3 and S4 to obtain the classification accuracy of the new model on the task data , if , then repeat S3-S5 to continue to increase the size of the neural network model; if and the number of growth in the depth direction of the neural network model is greater than the defined maximum growth number N, then stop the growth of the neural network model on the current task, and test based on the first task data of the test set to obtain the classification accuracy T1 of the model on the task data; S7: based on the second task data of the training set, repeating S3-S6 to obtain the classification accuracy T2 of the model on the task data; S8: repeating S7 until all image classifications of tasks are tested to obtain all classification accuracies of the self-grown neural network model on the test set, and evaluating the performance of the self-grown neural network model based on the classification accuracies.

2. The DNA damage repair mechanism inspired self-developing neural network design method of claim 1, wherein, The architecture of the neural network model in S1 is a convolutional neural network CNN or a linear layer Linear.

3. The DNA damage repair mechanism inspired self-developing neural network design method of claim 1, wherein, The sequence task division in S2 is determined based on the total number of categories of the image dataset and the number of learning categories of each task Specifically, the training set, the validation set and the test set are respectively divided into tasks, and the number of categories of each task is sequentially increased, that is, the first task has categories, the second task has categories, and the last task has categories.

4. The DNA damage repair mechanism inspired self-developing neural network design method of claim 1, wherein, For the neural network model of the CNN architecture, the growth in the width direction refers to increasing the number of convolution kernel channels, and the growth in the depth direction refers to increasing the number of convolution layers of the entire neural network model, the number of channels of the newly added convolution layer is set to half of the number of convolution kernel channels of the previous layer of the model, the last layer of the neural network model of the CNN architecture is a classification head, the size of the classification head is fixed and unchanged, and only the weight value is learned.

5. The DNA damage repair mechanism inspired self-developing neural network design method of claim 1, wherein, For the neural network model of the Linear architecture, the growth in the width direction refers to increasing the number of neurons in the same layer, and the growth in the depth direction refers to increasing the number of Linear layers of the entire neural network model, the number of neurons of the newly added Linear layer is set to half of the number of neurons of the previous layer of the model.

6. The DNA damage repair mechanism inspired self-developing neural network design method of claim 1, wherein, The implementation process of the neural network model spontaneous growth is divided into two steps: the first step is to grow a new network structure, and the second step is to merge the original network structure and the newly added network structure; for the neural network model of the CNN architecture, the growth parameter is , the new convolution channel number is first grown: , and then the new weight is spliced. For Linear architecture, the pre-growth parameters are , first grow the number of new neurons: , and then splice to get new weights .

7. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that when executed by a computer cause the computer to perform a method of any of claims 1-6. The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods of claims 1-6.

Images