Neural network training optimization method and system in small sample scenarios

Abstract

The application relates to the field of artificial intelligence, and discloses a neural network training optimization method and system in a small sample scene, which comprises the following steps: obtaining a pre-training classification neural network, a source domain reference sample set and a target small sample training set, and obtaining source domain reference features and target domain features corresponding to different feature levels of the pre-training classification neural network; determining feature distribution differences according to the source domain reference features and the target domain features corresponding to each feature level, and determining parameter freezing ranges and to-be-updated parameters of the corresponding feature levels according to the feature distribution differences; training the to-be-updated parameters; iteratively updating the to-be-updated parameters according to numerical calculation accuracy, obtaining a target classification neural network, and classifying to-be-classified samples by using the target classification neural network to obtain a category recognition result. The application scheme improves the migration adaptability, training efficiency and classification stability of a neural network model in a small sample scene.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for optimizing neural network training in small sample scenarios. Background Technology

[0002] With the development of intelligent manufacturing technology, neural network-based classification models are increasingly being applied to scenarios such as industrial inspection and equipment condition recognition. In actual industrial production, some inspection targets often have a low frequency of defects or limited sample collection conditions, resulting in a limited number of effective labeled samples available for model training. For example, in the defect detection of industrial parts, normal samples are easy to obtain, while defects such as cracks, wear, and abnormal structures are often sporadic, leading to a significant small-sample characteristic in the target detection task. To reduce sample collection costs, pre-trained classification neural networks combined with transfer learning are commonly used. This leverages existing model parameters from previous tasks to assist in training the target task, thereby improving the model's classification ability in small-sample scenarios.

[0003] However, in practical applications, different detection tasks have different data distributions, and the different levels of parameters in the same pre-trained classification neural network are not equally adapted to the target task. Existing transfer learning methods usually adopt fixed parameter freezing strategies or uniformly adjust the model parameter update range, which makes it difficult to adaptively adjust for the differences between the target small sample task and the original training task. This can easily lead to some parameters with transfer value being over-updated, or some parameters that need adjustment not being able to fully participate in training, thus affecting the training effect of the classification model.

[0004] Furthermore, during the training of few-sample classification models, the changes in different parameters vary at different training stages. Some parameters gradually stabilize during training, while others still require updates with high precision. Existing training methods typically use fixed computational precision for parameter updates, making it difficult to balance model training accuracy and computational resource consumption. Using high computational precision can easily lead to wasted computational resources, while using low computational precision may affect the stability of model training.

[0005] Therefore, how to reasonably control the range of model parameter updates based on the differences in target task features in small-sample classification tasks, and optimize the parameter update process in combination with parameter training status, so as to improve the training effect of classification models and reduce training resource consumption, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] This invention provides a method and system for optimizing neural network training in small sample scenarios, aiming to solve at least one of the above-mentioned technical problems.

[0007] To achieve the above objectives, this invention provides a method for optimizing neural network training in small-sample scenarios, comprising the following steps: Obtain a pre-trained classification neural network, a source domain reference sample set, and a target small sample training set, and obtain the source domain reference features and target domain features corresponding to different feature levels of the pre-trained classification neural network; The feature distribution differences are determined based on the source domain reference features and target domain features corresponding to each feature level, and the parameter freezing range and parameters to be updated for the corresponding feature level are determined based on the feature distribution differences. The parameters to be updated are trained based on the target small sample training set, and the corresponding numerical calculation accuracy is determined according to the gradient state information of the parameters to be updated. The parameters to be updated are iteratively updated according to the numerical calculation accuracy to obtain the target classification neural network, and the target classification neural network is used to classify the samples to be classified to obtain the category recognition result.

[0008] Optionally, a pre-trained classification neural network, a source domain reference sample set, and a target small sample training set are obtained, and source domain reference features and target domain features corresponding to different feature levels of the pre-trained classification neural network are obtained, including: Obtain the network layer structure information corresponding to the pre-trained classification neural network, and determine multiple feature levels based on the network layer structure information; The source domain reference sample set and the target small sample training set are respectively obtained and input into the pre-trained classification neural network. The output features corresponding to each feature level are then determined as the source domain reference features, and the output features corresponding to the target small sample training set are determined as the target domain features.

[0009] Optionally, the feature distribution difference is determined based on the source domain reference features and target domain features corresponding to each of the feature levels, including: For each of the aforementioned feature levels, the feature distribution distance between the corresponding source domain reference feature and the target domain feature is calculated to obtain the feature distribution distance corresponding to each of the aforementioned feature levels; The feature distribution difference corresponding to each feature level is determined based on the feature distribution distance corresponding to each feature level.

[0010] Optionally, determining the parameter freezing range and parameters to be updated for the corresponding feature level based on the feature distribution differences includes: Based on the differences in feature distribution corresponding to each feature level, determine the parameter freezing range corresponding to each feature level; According to the parameter freezing range corresponding to each feature level, the parameters of the corresponding feature level in the pre-trained classification neural network are frozen, and the unfrozen parameters are determined as the parameters to be updated for the corresponding feature level.

[0011] Optionally, based on the differences in feature distributions corresponding to each feature level, the parameter freezing range corresponding to each feature level is determined, including: The degree of migration matching corresponding to each feature level is determined based on the differences in feature distribution corresponding to each feature level. The proportion of pre-trained parameters retained in each feature level is determined based on the degree of migration matching, and the corresponding parameter freezing range is determined based on the proportion of pre-trained parameters retained.

[0012] Optionally, the parameters to be updated are trained based on the target small sample training set, and the corresponding numerical calculation precision is determined according to the gradient state information of the parameters to be updated, including: Based on the target small sample training set, the parameter to be updated is iteratively trained, and during the iterative training of the parameter to be updated, the gradient state information corresponding to the parameter to be updated is obtained. The gradient stability of the parameter to be updated is determined based on the gradient state information, and the numerical calculation accuracy of the parameter to be updated is determined based on the gradient stability.

[0013] Optionally, determining the gradient stability of the parameter to be updated based on the gradient state information, and determining the numerical calculation accuracy of the parameter to be updated based on the gradient stability, includes: Obtain the gradient state information corresponding to the parameter to be updated within a preset training period; Statistical analysis is performed on the gradient state information to determine the gradient change characteristics corresponding to the parameter to be updated; The gradient stability of the parameter to be updated is determined based on the gradient change characteristics. The accuracy level is matched according to the gradient stability, and the accuracy level is determined as the numerical calculation accuracy corresponding to the parameter to be updated.

[0014] Optionally, the parameters to be updated are iteratively updated according to the numerical calculation precision to obtain the target classification neural network, including: Configure the parameter update calculation method corresponding to the parameter to be updated according to the numerical calculation accuracy, and iteratively update the parameter to be updated based on the target small sample training set; During the iterative update process of the parameter to be updated, the gradient state information corresponding to the updated parameter to be updated is obtained; The numerical calculation precision corresponding to the parameter to be updated is re-determined based on the updated gradient state information corresponding to the parameter to be updated. The parameters to be updated are updated according to the newly determined numerical calculation accuracy until the preset training conditions are met, thereby obtaining the target classification neural network.

[0015] Optionally, the target classification neural network is used to classify the samples to be classified to obtain category recognition results, including: The sample to be classified is input into the target classification neural network, and the target classification neural network performs feature extraction and category matching on the sample to be classified to determine the category recognition result corresponding to the sample to be classified.

[0016] Furthermore, to achieve the above objectives, the present invention also provides a neural network training optimization system for small sample scenarios, comprising: The acquisition unit is used to acquire a pre-trained classification neural network, a source domain reference sample set, and a target small sample training set, and to acquire source domain reference features and target domain features corresponding to different feature levels of the pre-trained classification neural network. The update unit is used to determine the feature distribution difference based on the source domain reference features and target domain features corresponding to each feature level, and to determine the parameter freezing range and parameters to be updated for the corresponding feature level based on the feature distribution difference. The training unit is used to train the parameters to be updated based on the target small sample training set, and to determine the corresponding numerical calculation accuracy according to the gradient state information of the parameters to be updated. The execution unit is used to iteratively update the parameters to be updated according to the numerical calculation accuracy, obtain the target classification neural network, and use the target classification neural network to classify the samples to be classified to obtain the category recognition result.

[0017] The beneficial effects of this invention are as follows: By analyzing the differences in feature distribution corresponding to different feature levels, the parameter freezing range is dynamically determined, making the parameter updates in the pre-trained classification neural network more aligned with the requirements of the target small-sample task, thus improving the model transfer training effect. Simultaneously, by adjusting the numerical calculation precision based on the gradient state information of the parameters to be updated, the training process adapts to parameter changes, reducing unnecessary computational resource consumption and improving the training efficiency and stability of the small-sample classification model. This method is suitable for small-sample classification scenarios such as industrial inspection, and can improve the model's classification ability under limited sample conditions. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention; Figure 2This is a flowchart illustrating an embodiment of the neural network training optimization method for small sample scenarios of the present invention. Figure 3 This is a structural block diagram of a neural network training and optimization system in a small sample scenario according to an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] like Figure 1 As shown, Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.

[0021] like Figure 1 As shown, the device may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or stable non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0022] Those skilled in the art will understand that Figure 1 The structure of the device shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0023] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an acquisition unit, an update unit, a training unit, an execution unit, and a neural network training and optimization program for small sample scenarios.

[0024] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; while the processor 1001 can be used to call the neural network training and optimization program for small sample scenarios stored in the memory 1005.

[0025] This invention provides a method for optimizing neural network training in a small-sample scenario, referring to... Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the neural network training optimization method for small sample scenarios according to the present invention.

[0026] In this embodiment, a method for optimizing neural network training in a small sample scenario includes the following steps: S100: Obtain a pre-trained classification neural network, a source domain reference sample set, and a target small sample training set, and obtain the source domain reference features and target domain features corresponding to different feature levels of the pre-trained classification neural network.

[0027] Step S200: Determine the feature distribution difference based on the source domain reference features and target domain features corresponding to each feature level, and determine the parameter freezing range and parameters to be updated for the corresponding feature level based on the feature distribution difference.

[0028] Step S300: Train the parameters to be updated based on the target small sample training set, and determine the corresponding numerical calculation accuracy according to the gradient state information of the parameters to be updated.

[0029] Step S400: Iteratively update the parameters to be updated according to the numerical calculation accuracy to obtain the target classification neural network, and use the target classification neural network to classify the samples to be classified to obtain the category recognition result.

[0030] Specifically, in step S100, obtaining a pre-trained classification neural network, a source domain reference sample set, and a target small sample training set, and obtaining source domain reference features and target domain features corresponding to different feature levels of the pre-trained classification neural network, includes: obtaining network layer structure information corresponding to the pre-trained classification neural network, determining multiple feature levels based on the network layer structure information; obtaining the output features corresponding to each feature level after the source domain reference sample set and the target small sample training set are input into the pre-trained classification neural network, determining the output features corresponding to the source domain reference sample set as the source domain reference features, and determining the output features corresponding to the target small sample training set as the target domain features.

[0031] In this embodiment, the network layer structure information corresponding to the pre-trained classification neural network is obtained, and multiple feature levels in the pre-trained classification neural network are determined based on the network layer structure information.

[0032] Specifically, feature levels are defined based on the feature extraction tasks performed by each network layer in the pre-trained classification neural network. Network layers that continuously perform low-level feature extraction are designated as low-level feature levels; network layers located after the low-level feature levels and used for feature combination and abstract expression are designated as mid-level feature levels; and network layers near the classification output and used for category semantic representation are designated as high-level feature levels. For example, when using a convolutional neural network as the pre-trained classification neural network, the initial convolutional layers, mid-level convolutional layers, and convolutional layers near the fully connected classification layer are designated as different feature levels, and feature level identifiers, such as Layer_Level_1, Layer_Level_2, and Layer_Level_3, are established to associate the feature information output by the corresponding network layers.

[0033] Furthermore, a source domain reference sample set and a target small sample training set are obtained respectively. The sample data in the source domain reference sample set and the target small sample training set are input into the pre-trained classification neural network according to a preset input format, so that the pre-trained classification neural network extracts features sequentially based on each network layer.

[0034] The source domain reference sample set is the set of samples corresponding to the original training task of the pre-trained classification neural network, containing historical sample data with category labels; the target small sample training set is a small amount of labeled sample data corresponding to the classification task to be adapted. For example, in the scenario of industrial component defect classification, the source domain reference sample set includes normal samples and defective samples corresponding to existing product types, and the target small sample training set includes a small amount of defective samples corresponding to new product models.

[0035] The output features of the source domain reference sample set and the target small sample training set after passing through each feature level are obtained respectively. The output features are associated and stored according to the feature level identification information. The output features corresponding to the source domain reference sample set are determined as source domain reference features, and the output features corresponding to the target small sample training set are determined as target domain features.

[0036] For example, for the i-th feature level, the feature output by the j-th sample in the source domain reference sample set after passing through this feature level is represented as the source domain reference feature, and the feature output by the j-th sample in the target small sample training set after passing through the same feature level is represented as the target domain feature. The correspondence is established by means of "sample number + feature level number", so that the feature distribution difference can be calculated for the source domain reference feature and the target domain feature corresponding to the same feature level in the future.

[0037] It should be noted that the feature level division method is not limited to the convolutional layer division method. Depending on the network structure of the pre-trained classification neural network, the corresponding feature levels are divided according to the network units in the network that actually undertake the functions of feature extraction, feature fusion, and category semantic expression. For example, for a network structure containing multiple encoding modules, the network modules corresponding to different encoding stages are determined as different feature levels, thereby ensuring that the obtained source domain reference features and target domain features can reflect the feature expression differences at different depths of the pre-trained classification neural network.

[0038] Specifically, in step S200, determining the feature distribution difference based on the source domain reference feature and target domain feature corresponding to each feature level includes: for each feature level, calculating the feature distribution distance between the corresponding source domain reference feature and target domain feature to obtain the feature distribution distance corresponding to each feature level; and determining the feature distribution difference corresponding to each feature level based on the feature distribution distance corresponding to each feature level.

[0039] Further, determining the parameter freezing range and parameters to be updated for the corresponding feature level based on the feature distribution differences includes: determining the parameter freezing range for each feature level based on the feature distribution differences for each feature level; freezing the parameters of the corresponding feature level in the pre-trained classification neural network according to the parameter freezing range for each feature level, and determining the unfrozen parameters as the parameters to be updated for the corresponding feature level.

[0040] Further, based on the feature distribution differences corresponding to each feature level, the parameter freezing range corresponding to each feature level is determined, including determining the transfer matching degree corresponding to each feature level based on the feature distribution differences corresponding to each feature level; determining the proportion of pre-trained parameters retained in each feature level based on the transfer matching degree; and determining the corresponding parameter freezing range based on the proportion of pre-trained parameters retained.

[0041] In this embodiment, step S200 involves determining the feature distribution difference based on the source domain reference features and target domain features corresponding to each feature level, and determining the parameter freezing range and parameters to be updated for the corresponding feature level based on the feature distribution difference, including: For each of the aforementioned feature levels, the feature distribution distance between the corresponding source domain reference feature and the target domain feature is calculated to obtain the feature distribution distance corresponding to each of the aforementioned feature levels.

[0042] Specifically, for the first The first feature level is obtained. The source domain reference feature set corresponding to each feature level and target domain feature set The source domain reference feature set refers to... This indicates that the source domain reference sample set has been processed by the pre-trained classification neural network. The feature set obtained after processing each feature level, the target domain feature set This indicates that the target small sample training set has been processed by the pre-trained classification neural network. The feature set obtained after processing each feature level.

[0043] Regarding the first The source domain reference feature set is calculated using the maximum mean difference method at each feature level. With the target domain feature set The distribution distance between them is used to obtain the first... Feature distribution distance corresponding to each feature level ,in: ; in, Indicates the first The feature distribution distance corresponding to each feature level; Indicates the number of source domain reference features; Indicates the number of features in the target domain; Indicates the first The first feature level corresponding to the first Each source domain reference feature; Indicates the first The first feature level corresponding to the first Each target domain feature; This represents a mapping function that maps features to a high-dimensional feature space.

[0044] Furthermore, the corresponding feature distribution differences are determined based on the feature distribution distances corresponding to each of the aforementioned feature levels.

[0045] Specifically, since there are differences in the number of features, feature dimensions, and feature expression range corresponding to different feature levels, in order to improve the comparison consistency between different feature levels, the feature distribution distance corresponding to each feature level is normalized to obtain the corresponding feature distribution differences.

[0046] The output features of each layer are first processed by uniform pooling and normalization, and the same kernel function and uniform kernel parameter determination rules are used before calculating MMD. For a pre-trained classification neural network containing L feature levels, the feature distribution difference corresponding to the i-th feature level is expressed as: ; in, Indicates the first The feature distribution differences corresponding to each feature level; L represents the number of feature levels obtained by the pre-trained classification neural network. Indicates the first The feature distribution distances corresponding to each feature level are used to reduce the impact of feature scales at different feature levels on distance comparisons.

[0047] For example, in the scenario of industrial component defect classification, the pre-trained classification neural network includes three feature levels, which are used to extract surface texture features, structural combination features, and semantic features of defect categories, respectively. After calculating the feature distribution distance, the following values ​​are obtained: D1=0.12; D2=0.25; D3=0.41.

[0048] The corresponding feature distribution differences are as follows: ; ; ; This determines the degree of distribution variation between the source domain reference features and the target domain features corresponding to different feature levels.

[0049] Furthermore, the parameter freezing range and parameters to be updated for the corresponding feature level are determined based on the differences in feature distribution.

[0050] Specifically, the retention ratio of pre-training parameters for each feature level is determined based on the differences in feature distributions corresponding to each feature level.

[0051] Specifically, the retention level of pre-trained parameters in each feature level is adjusted based on the feature distribution differences corresponding to each feature level. For feature levels with smaller feature distribution differences, it indicates that the features extracted by this feature level have high consistency between the source domain task and the target few-sample task, thus increasing the parameter freezing range corresponding to this feature level. For feature levels with larger feature distribution differences, it indicates that this feature level needs more adaptation for the target few-sample task, thus decreasing the corresponding parameter freezing range. In specific implementation, the network sub-layer or parameter tensor in the corresponding feature level is used as the parameter freezing unit, and the corresponding network sub-layer or parameter tensor is frozen according to the determined parameter freezing range, while the remaining unfrozen parameters are determined as parameters to be updated.

[0052] Furthermore, based on the parameter freezing range corresponding to each feature level, the parameters of the corresponding feature level in the pre-trained classification neural network are frozen.

[0053] Specifically, according to the determined parameter freezing range, the pre-training parameters in the corresponding feature level that do not need to participate in the target few-sample training are frozen, so that the frozen parameters remain unchanged in the subsequent training process; at the same time, the unfrozen parameters are determined as the parameters to be updated in the corresponding feature level, and the parameters to be updated are adjusted using the target few-sample training set.

[0054] For example, in the task of classifying defects in industrial parts, after the pre-trained classification neural network is trained based on existing defect samples, when it is transferred to the task of identifying defects in new model parts, if the feature distribution difference corresponding to the feature level used to extract surface texture features is small, it indicates that the historical parts and the new model parts have similar surface texture expressions. In this case, more pre-trained parameters in the feature level are retained, and only a few parameters are adjusted. If the feature distribution difference corresponding to the feature level used to express the semantics of the defect category is large, it indicates that the original defect category expression is significantly different from the target task. In this case, more parameters are released to participate in the training to improve the model's adaptability to new defect categories.

[0055] By employing the above method, based on the distribution differences between the source domain reference features and the target domain features corresponding to different feature levels, a differentiated freezing strategy is implemented for the parameters at different levels in the pre-trained classification neural network. This allows pre-trained parameters with transfer value to be retained, while parameters that differ significantly from the target small sample task are fully involved in training. This reduces ineffective parameter adjustments during small sample training and improves the transfer adaptability of the target classification neural network.

[0056] Preferably, since there is a feature transfer relationship between adjacent feature layers in the pre-trained classification neural network, although some shallow feature parameters have small differences in their own feature distribution, their output features can affect the expression of subsequent high-level semantic features. Therefore, when determining the parameter freezing range, adjacent feature layers are determined according to the forward feature transfer order of the pre-trained classification neural network, and the trend of feature distribution differences between adjacent feature layers is compared.

[0057] For example, in the scenario of industrial component defect classification, the first feature level corresponds to surface texture features, and the second feature level corresponds to defect structure features. If the difference in feature distribution between the first feature level and the second feature level is small, the continuity of the frozen range between the two feature levels is maintained to avoid unstable feature transmission caused by local parameter adjustments. If the difference between two adjacent feature levels is significant, the freezing degree corresponding to the high-difference feature level is reduced to enable it to gain more target task adaptability.

[0058] By using the above method, based on determining the parameter freezing range based on the differences in feature distribution, we further consider the correlation between different feature levels, improve the matching degree between the parameter freezing strategy and the feature transmission law inside the neural network, and avoid the problem of parameter adjustment inconsistency caused by relying solely on the differences in single-layer features.

[0059] Specifically, in step S300, training the parameter to be updated based on the target small sample training set and determining the corresponding numerical calculation accuracy based on the gradient state information of the parameter to be updated includes: iteratively training the parameter to be updated based on the target small sample training set, obtaining the gradient state information corresponding to the parameter to be updated during the iterative training process; determining the gradient stability of the parameter to be updated based on the gradient state information, and determining the numerical calculation accuracy of the parameter to be updated based on the gradient stability.

[0060] Further, determining the gradient stability of the parameter to be updated based on the gradient state information, and determining the numerical calculation accuracy of the parameter to be updated based on the gradient stability, includes: acquiring the gradient state information of the parameter to be updated within a preset training period; performing statistical analysis on the gradient state information to determine the gradient change characteristics of the parameter to be updated; determining the gradient stability of the parameter to be updated based on the gradient change characteristics; matching the corresponding accuracy level based on the gradient stability, and determining the accuracy level as the numerical calculation accuracy of the parameter to be updated.

[0061] In this embodiment of the invention, during the iterative training of the parameter to be updated using the target small sample training set, the gradient state information corresponding to the parameter to be updated is obtained.

[0062] Specifically, the parameters to be updated are the unfrozen parameters determined in step S200, which participate in parameter adjustment during the target small sample training process. Based on the network sublayer and parameter tensor to which the parameters to be updated belong, the parameters belonging to the same parameter tensor are grouped into the same parameter group. During each parameter update, the gradient state information corresponding to each parameter group is obtained, and a gradient state recording sequence is established according to the training iteration order. The gradient state recording sequence is used to reflect the gradient change magnitude and trend of the corresponding parameter group during continuous training.

[0063] Furthermore, statistical analysis is performed on the gradient state information to determine the gradient change characteristics corresponding to the parameter to be updated.

[0064] Specifically, gradient state information is acquired during multiple consecutive training iterations within a preset training period, and the gradient changes during adjacent training iterations are analyzed to determine the gradient change characteristics corresponding to the parameter to be updated.

[0065] The gradient change feature is used to characterize the stability of the parameter to be updated during the training process, including at least one of gradient change magnitude, gradient fluctuation degree, and gradient change trend.

[0066] For example, during the training of an industrial component defect classification model, gradient state information is continuously acquired over 50 training iterations for the parameters to be updated in the training of defect classification for new component models. When the gradient magnitude changes little and the gradient direction remains consistent over multiple consecutive training cycles, it indicates that the parameter to be updated has gradually adapted to the target small sample task; when the gradient magnitude changes continuously or the gradient direction changes frequently, it indicates that the parameter to be updated is still in the rapid adjustment phase.

[0067] Furthermore, the gradient stability of the parameter to be updated is determined based on the gradient change characteristics.

[0068] Specifically, the current training state of the parameter to be updated is determined based on the gradient change characteristics. When the gradient change amplitude corresponding to the parameter to be updated is small and the gradient change trend remains stable within consecutive training periods, the gradient stability of the parameter to be updated is determined to be high; when the gradient change amplitude corresponding to the parameter to be updated is large, or the gradient change trend fluctuates significantly within consecutive training periods, the gradient stability of the parameter to be updated is determined to be low.

[0069] For example, a training cycle is defined as 10 consecutive training iterations. The average gradient norm of the corresponding parameter group to be updated is calculated within each training cycle, and the degree of change in the average gradient norm between adjacent training cycles is compared. States where the rate of change in the average gradient norm between adjacent training cycles is less than 5% are defined as stable states, states where the rate of change is greater than 20% are defined as unstable states, and states where the rate of change is between 5% and 20% are defined as transitional states. After maintaining the same state for two consecutive training cycles, the corresponding gradient stability is updated to avoid frequent switching of accuracy levels caused by gradient fluctuations in a single training cycle. This method distinguishes the training states of different parameters to be updated.

[0070] Furthermore, a corresponding accuracy level is matched based on the gradient stability, and the accuracy level is determined as the numerical calculation accuracy corresponding to the parameter to be updated.

[0071] Specifically, the precision of the parameter update calculation corresponding to the parameter to be updated is adjusted based on the gradient stability. When the gradient stability of the parameter to be updated is high, it indicates that the parameter to be updated is close to the target task adaptation state, and the sensitivity to parameter change precision is reduced. Therefore, a lower level of numerical calculation precision is used for parameter update. When the gradient stability of the parameter to be updated is low, it indicates that the parameter to be updated is still in a rapid adjustment phase, and a higher level of numerical calculation precision needs to be maintained to ensure the stability of the parameter update process.

[0072] For example, numerical computation precision can be divided into high-precision, medium-precision, and low-precision levels. High-precision corresponds to 32-bit floating-point computation, medium-precision corresponds to 16-bit floating-point computation, and low-precision corresponds to 8-bit floating-point computation. When the parameter set to be updated is in an unstable state, 32-bit floating-point precision is used; when it is in a transitional state, 16-bit floating-point precision is used; and when it remains stable for two consecutive training epochs, 8-bit floating-point precision is used. When using 16-bit or 8-bit floating-point precision, the master parameters and optimizer state in 32-bit floating-point format are retained, and the reduced numerical computation precision is used for forward computation, backward computation, and gradient tensor processing.

[0073] It should be noted that the statistical period of the gradient state information, the specific evaluation method of gradient change characteristics, and the classification method of numerical calculation accuracy levels are configured according to the target classification neural network structure and training resource conditions. This embodiment dynamically determines the numerical calculation accuracy based on the actual changes of the parameters to be updated during the training process, ensuring that the parameters to be updated at different training stages adopt a calculation accuracy that matches their training state. This avoids the waste of computational resources or reduced training stability caused by using a fixed calculation accuracy throughout the training process. This method is suitable for small-sample model training scenarios such as industrial component defect classification.

[0074] Preferably, during the training of the target small sample, the gradient state information corresponding to multiple parameters to be updated within the same feature level is obtained, and it is determined whether the gradient change trends of different parameters to be updated are consistent. When the gradient state change trends of multiple parameters to be updated are similar, it indicates that the overall training state of the feature level tends to be stable, and the corresponding parameters are updated with the same precision level; when there are significant differences in the gradient change trends of different parameters to be updated within the same feature level, the numerical calculation precision is determined according to the gradient stability of each parameter to be updated.

[0075] For example, during the training of an industrial component defect classification model, for multiple parameters to be updated in the semantic feature level of defect categories, the gradient change rate of some weight parameters has decreased to a stable range, while some bias parameters still have large fluctuations. Therefore, the weight parameters are kept to be calculated using a low-precision method, while the bias parameters are kept to be calculated using a high-precision method for continued updates.

[0076] By using the above method, based on adjusting the numerical calculation precision according to the gradient state of a single parameter to be updated, the training differences between parameters within the same feature level are further considered, so that different parameters can be matched with more refined calculation precision, reducing the waste of computing resources caused by uniform precision configuration, while ensuring the training stability during the rapid parameter change stage.

[0077] Specifically, in step S400, iteratively updating the parameters to be updated according to the numerical calculation precision to obtain the target classification neural network includes: configuring the parameter update calculation method corresponding to the parameters to be updated according to the numerical calculation precision, and iteratively updating the parameters to be updated based on the target small sample training set; during the iterative update of the parameters to be updated, obtaining the gradient state information corresponding to the updated parameters to be updated; redetermining the numerical calculation precision corresponding to the updated parameters to be updated according to the gradient state information corresponding to the updated parameters to be updated; and continuing to update the parameters to be updated according to the redetermined numerical calculation precision until the preset training conditions are met to obtain the target classification neural network.

[0078] Furthermore, the target classification neural network is used to classify the sample to be classified and obtain the category recognition result, including: inputting the sample to be classified into the target classification neural network, performing feature extraction and category matching on the sample to be classified through the target classification neural network, and determining the category recognition result corresponding to the sample to be classified.

[0079] In this embodiment of the invention, based on the numerical calculation precision corresponding to the parameters to be updated determined in step S300, the precision of the forward calculation, backward calculation, and gradient tensor processing corresponding to the parameter group to be updated is configured, and the master parameters and optimizer state in 32-bit floating-point format are retained. After completing the corresponding training iteration, the master parameters are updated using the current gradient, so that the parameter group to be updated under different training states participates in training with the corresponding numerical calculation precision. For example, when the gradient state corresponding to the parameter to be updated is in a rapidly changing stage, the corresponding parameter update calculation method is configured as a high-precision calculation method; when the gradient state corresponding to the parameter to be updated gradually stabilizes, the corresponding parameter update calculation method is adjusted to a low-precision calculation method to reduce the consumption of computational resources in the stable training stage.

[0080] Furthermore, the parameters to be updated are iteratively updated based on the target small sample training set. During each parameter update, the gradient calculation and parameter adjustment corresponding to the parameters to be updated are completed according to the currently determined numerical calculation accuracy, and the parameter state of the corresponding feature layer in the pre-trained classification neural network is updated.

[0081] For example, in the scenario of industrial component defect classification, a pre-trained classification neural network with frozen parameters is applied to the task of identifying defects in new model components. A small number of labeled defect samples are used as the target small sample training set to iteratively optimize the unfrozen parameters to be updated. In the initial training phase, the parameters to be updated vary greatly, and 32-bit floating-point precision is used to update the parameters. As the training process progresses, some parameters to be updated gradually stabilize, and the updated parameters are adjusted to 16-bit floating-point precision or 8-bit precision based on the redefined numerical calculation precision.

[0082] Furthermore, during the iterative update process of the parameter to be updated, the gradient state information corresponding to the updated parameter to be updated is obtained, and the numerical calculation precision corresponding to the parameter to be updated is re-determined based on the updated gradient state information.

[0083] Specifically, after completing a preset number of parameter updates, the gradient state information corresponding to the parameter to be updated is re-collected, the gradient change after the update is analyzed, it is determined whether the current parameter to be updated is still in a rapid change phase, and the numerical calculation accuracy corresponding to the next training phase is adjusted according to the judgment result, thereby forming a closed-loop process in which the numerical calculation accuracy is dynamically adjusted according to the parameter training state.

[0084] Furthermore, the parameters to be updated are updated according to the newly determined numerical calculation precision until the preset training conditions are met, thereby obtaining the target classification neural network.

[0085] Specifically, the target small sample training set is divided into a training subset and a validation subset according to a preset ratio. The training subset is used for iterative updates of the parameters to be updated, and the validation subset is used to determine the training convergence status. For example, the training subset and the validation subset are divided in an 8:2 ratio. If the improvement in the classification loss of the validation subset is less than 1% within five consecutive training cycles, it is determined that a preset convergence condition has been met, and 100 rounds are set as the maximum number of training iterations. When the preset convergence condition is met or the maximum number of training iterations is reached, the updating of the parameters to be updated is stopped, and the pre-trained classification neural network after parameter optimization is determined as the target classification neural network.

[0086] Furthermore, the target classification neural network is used to classify the samples to be classified to obtain category recognition results, including: The sample to be classified is input into the target classification neural network, and the target classification neural network performs feature extraction and category matching on the sample to be classified to determine the category recognition result corresponding to the sample to be classified.

[0087] Specifically, the target classification neural network extracts features layer by layer from the input samples to be classified, obtains the feature representations corresponding to the samples, and determines the category recognition result based on the matching relationship between the feature representations and preset category information. For example, in an industrial component defect detection scenario, the image of the component to be detected is input into the target classification neural network, and after feature extraction, the corresponding defect category is output, including category recognition results such as crack defects, wear defects, or normal state.

[0088] It should be noted that the method for adjusting the numerical calculation precision is not limited to the aforementioned precision level classification. Any other precision representation method used, provided it satisfies the requirement of dynamically adjusting the parameter update precision based on the gradient state of the parameter to be updated, falls within the scope of this solution. Through this method, the target classification neural network balances model adaptability and computational resource utilization efficiency during small-sample training, thereby improving the stability of the target classification neural network in practical classification tasks.

[0089] In one specific embodiment, the neural network training and optimization method in the small sample scenario is applied to the surface defect classification scenario of industrial parts, and is used to classify and identify surface defects of new bearing parts.

[0090] Obtain surface inspection images of existing bearing components as the source domain reference sample set, and establish sample labels for each sample. For example: SRC-BEAR-A001-NORMAL SRC-BEAR-A002-CRACK SRC-BEAR-A003-SCRATCH In this context, SRC represents the source domain sample, BEAR represents the bearing component type, and the following characters represent the sample number and defect category.

[0091] For new bearing components, a small number of labeled detection images were collected as a target small sample training set, and a target domain sample label was established: TGT-BEAR-B001-CRACK TGT-BEAR-B002-WEAR Obtain a pre-trained classification neural network trained on a source domain reference sample set, divide it into three feature levels according to the network structure, and establish feature level labels: FEATURE_LEVEL_01 FEATURE_LEVEL_02 FEATURE_LEVEL_03 Among them, FEATURE_LEVEL_01 is used to extract surface texture features, FEATURE_LEVEL_02 is used to extract defect structure features, and FEATURE_LEVEL_03 is used to extract defect category semantic features.

[0092] The source domain reference sample set and the target small sample training set are respectively input into the pre-trained classification neural network to obtain the source domain reference features and target domain features corresponding to each feature level, and establish the correspondence, for example: SRC-BEAR-A001-CRACK FEATURE_LEVEL_01 SRC_FEATURE_VECTOR_001 as well as: TGT-BEAR-B001-CRACK FEATURE_LEVEL_01 TGT_FEATURE_VECTOR_001 Furthermore, the distribution differences between the source domain reference features and the target domain features corresponding to each feature level are calculated to obtain: FEATURE_LEVEL_01_DISTANCE=0.12 FEATURE_LEVEL_02_DISTANCE=0.28 FEATURE_LEVEL_03_DISTANCE=0.60 The parameter freezing range is determined based on the differences in the aforementioned feature distributions. Specifically, the smaller differences at the first feature level indicate high consistency in texture features between the source and target domains, allowing for the retention of more pre-trained parameters. Conversely, the larger differences at the third feature level indicate significant semantic changes in the defect category, reducing the number of frozen parameters and enabling more parameters to participate in the target small-sample training.

[0093] Create a set of parameters to be updated from the unfrozen parameters: UPDATE_PARAMETER_GROUP_SET={LAYER03_WEIGHT_TENSOR,LAYER03_BIAS_TENSOR} TRAIN_STEP=0010 PARAM_GROUP_ID=LAYER03_WEIGHT_TENSOR GRADIENT_STATE=NON_STABLEPRECISION_MODE=FP32 The parameters to be updated are iteratively updated based on the target small sample training set, and the gradient state is recorded during the training process. For example: TRAIN_STEP=0050 PARAM_GROUP_ID=LAYER03_WEIGHT_TENSOR GRADIENT_STATE=TRANSITION PRECISION_MODE=FP16 This indicates that the current parameter is in a rapid adjustment phase, using: PRECISION_MODE=FP32 Update the parameters.

[0094] When the gradient changes gradually stabilize during training, adjust as follows: TRAIN_STEP=0050 PARAM_ID=PARAM_LAYER03_WEIGHT_001 GRADIENT_STATE=STABLE PRECISION_MODE=FP16 Continue updating the parameters. Once the preset training conditions are met, generate the target classification neural network: MODEL_ID=TGT_BEAR_DEFECT_MODEL_V1 TRAIN_STATUS=COMPLETED Finally, the image of the component to be classified is input into the target classification neural network, which outputs the category recognition result. For example: IMAGE_ID=TEST-BEAR-C001 RESULT_CLASS=CRACK This indicates that the sample to be classified has been identified as a crack defect.

[0095] Through the above embodiments, in the scenario of industrial component defect classification, the parameter freezing range is adjusted according to the distribution differences of different feature levels, and the calculation accuracy is dynamically adjusted in combination with the gradient state during the training process. This enables the target classification neural network to complete rapid adaptation with a small number of samples, thereby improving the training efficiency and recognition stability in small sample classification tasks.

[0096] Reference Figure 3, Figure 3 This is a structural block diagram of an embodiment of the neural network training and optimization system for small sample scenarios of the present invention.

[0097] like Figure 3 As shown, the neural network training optimization system for small sample scenarios proposed in this embodiment of the invention includes: The acquisition unit is used to acquire a pre-trained classification neural network, a source domain reference sample set, and a target small sample training set, and to acquire source domain reference features and target domain features corresponding to different feature levels of the pre-trained classification neural network. The update unit is used to determine the feature distribution difference based on the source domain reference features and target domain features corresponding to each feature level, and to determine the parameter freezing range and parameters to be updated for the corresponding feature level based on the feature distribution difference. The training unit is used to train the parameters to be updated based on the target small sample training set, and to determine the corresponding numerical calculation accuracy according to the gradient state information of the parameters to be updated. The execution unit is used to iteratively update the parameters to be updated according to the numerical calculation accuracy, obtain the target classification neural network, and use the target classification neural network to classify the samples to be classified to obtain the category recognition result.

[0098] It is understood that in the description of this specification, references to terms such as "one embodiment," "another embodiment," "other embodiments," or "first embodiment to Nth embodiment," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0099] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0100] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0101] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0102] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for optimizing neural network training in a few-sample scenario, characterized in that, Includes the following steps: Obtain a pre-trained classification neural network, a source domain reference sample set, and a target small sample training set, and obtain the source domain reference features and target domain features corresponding to different feature levels of the pre-trained classification neural network; The feature distribution differences are determined based on the source domain reference features and target domain features corresponding to each feature level, and the parameter freezing range and parameters to be updated for the corresponding feature level are determined based on the feature distribution differences. The parameters to be updated are trained based on the target small sample training set, and the corresponding numerical calculation accuracy is determined according to the gradient state information of the parameters to be updated. The parameters to be updated are iteratively updated according to the numerical calculation accuracy to obtain the target classification neural network, and the target classification neural network is used to classify the samples to be classified to obtain the category recognition result.

2. The neural network training optimization method in a small sample scenario as described in claim 1, characterized in that, Obtain a pre-trained classification neural network, a source domain reference sample set, and a target small sample training set, and obtain the source domain reference features and target domain features corresponding to different feature levels of the pre-trained classification neural network, including: Obtain the network layer structure information corresponding to the pre-trained classification neural network, and determine multiple feature levels based on the network layer structure information; The source domain reference sample set and the target small sample training set are respectively obtained and input into the pre-trained classification neural network. The output features corresponding to each feature level are then determined as the source domain reference features, and the output features corresponding to the target small sample training set are determined as the target domain features.

3. The neural network training optimization method in a small sample scenario as described in claim 1, characterized in that, The feature distribution differences are determined based on the source domain reference features and target domain features corresponding to each feature level, including: For each of the aforementioned feature levels, the feature distribution distance between the corresponding source domain reference feature and the target domain feature is calculated to obtain the feature distribution distance corresponding to each of the aforementioned feature levels; The feature distribution difference corresponding to each feature level is determined based on the feature distribution distance corresponding to each feature level.

4. The neural network training optimization method in a small sample scenario as described in claim 3, characterized in that, Based on the differences in feature distribution, the parameter freezing range and parameters to be updated for the corresponding feature level are determined, including: Based on the differences in feature distribution corresponding to each feature level, determine the parameter freezing range corresponding to each feature level; According to the parameter freezing range corresponding to each feature level, the parameters of the corresponding feature level in the pre-trained classification neural network are frozen, and the unfrozen parameters are determined as the parameters to be updated for the corresponding feature level.

5. The neural network training optimization method in a small sample scenario as described in claim 4, characterized in that, Based on the feature distribution differences corresponding to each feature level, determine the parameter freezing range corresponding to each feature level, including: The degree of migration matching corresponding to each feature level is determined based on the differences in feature distribution corresponding to each feature level. The proportion of pre-trained parameters retained in each feature level is determined based on the degree of migration matching, and the corresponding parameter freezing range is determined based on the proportion of pre-trained parameters retained.

6. The neural network training optimization method in a small sample scenario as described in claim 1, characterized in that, The parameters to be updated are trained based on the target small sample training set, and the corresponding numerical calculation accuracy is determined according to the gradient state information of the parameters to be updated, including: Based on the target small sample training set, the parameter to be updated is iteratively trained, and during the iterative training of the parameter to be updated, the gradient state information corresponding to the parameter to be updated is obtained. The gradient stability of the parameter to be updated is determined based on the gradient state information, and the numerical calculation accuracy of the parameter to be updated is determined based on the gradient stability.

7. The neural network training optimization method in a small sample scenario as described in claim 6, characterized in that, Determine the gradient stability of the parameter to be updated based on the gradient state information, and determine the numerical calculation accuracy of the parameter to be updated based on the gradient stability, including: Obtain the gradient state information corresponding to the parameter to be updated within a preset training period; Statistical analysis is performed on the gradient state information to determine the gradient change characteristics corresponding to the parameter to be updated; The gradient stability of the parameter to be updated is determined based on the gradient change characteristics. The accuracy level is matched according to the gradient stability, and the accuracy level is determined as the numerical calculation accuracy corresponding to the parameter to be updated.

8. The neural network training optimization method in a small sample scenario as described in claim 1, characterized in that, The parameters to be updated are iteratively updated according to the numerical calculation accuracy to obtain the target classification neural network, including: Configure the parameter update calculation method corresponding to the parameter to be updated according to the numerical calculation accuracy, and iteratively update the parameter to be updated based on the target small sample training set; During the iterative update process of the parameter to be updated, the gradient state information corresponding to the updated parameter to be updated is obtained; The numerical calculation precision corresponding to the parameter to be updated is re-determined based on the updated gradient state information corresponding to the parameter to be updated. The parameters to be updated are updated according to the newly determined numerical calculation accuracy until the preset training conditions are met, thereby obtaining the target classification neural network.

9. The neural network training optimization method in a small sample scenario as described in claim 8, characterized in that, The target classification neural network is used to classify the samples to be classified and obtain the category recognition result, including: The sample to be classified is input into the target classification neural network, and the target classification neural network performs feature extraction and category matching on the sample to be classified to determine the category recognition result corresponding to the sample to be classified.

10. A neural network training and optimization system for small sample scenarios, characterized in that, include: The acquisition unit is used to acquire a pre-trained classification neural network, a source domain reference sample set, and a target small sample training set, and to acquire source domain reference features and target domain features corresponding to different feature levels of the pre-trained classification neural network. The update unit is used to determine the feature distribution difference based on the source domain reference features and target domain features corresponding to each feature level, and to determine the parameter freezing range and parameters to be updated for the corresponding feature level based on the feature distribution difference. The training unit is used to train the parameters to be updated based on the target small sample training set, and to determine the corresponding numerical calculation accuracy according to the gradient state information of the parameters to be updated. The execution unit is used to iteratively update the parameters to be updated according to the numerical calculation accuracy, obtain the target classification neural network, and use the target classification neural network to classify the samples to be classified to obtain the category recognition result.

Images