Training method and device of neural network based on time self-erasing

Abstract

Provided are a training method and device of a time self-erasing neural network. The training method comprises: for each time step, generating a first feature map corresponding to the time step by inputting a training sample into a convolutional layer; for each time step except the first time step, generating a second feature map by inputting the first feature map to which an erasing mask varying with each time step is applied into a pooling layer; generating a first prediction map by inputting the second feature map into a first fully connected layer; generating a loss based on the first prediction map and a real label; and obtaining a trained neural network by adjusting parameters of the neural network based on the loss. Thus, the overall prediction accuracy is improved, the calculation efficiency is improved, and the calculation overhead is reduced.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to training of neural networks, and more particularly, to a method and apparatus for training of a time-erasing-based neural network. BACKGROUND

[0002] In the prior art, a spiking neural network (SNN), such as a biologically plausible model of neural networks, has attracted extensive attention in recent years. Compared with a traditional artificial neural network (ANN), SNN is able to simulate the dynamic process of biological neurons by delivering information in multiple time steps through spiking signals.

[0003] However, although SNN has the advantage of dynamic inference, they still have a bottleneck in feature discrimination ability, mainly because the traditional training method usually produces the same backpropagation gradient, lacking diversity in feature representation. This problem makes SNN often focus on the same area of multimedia data such as images and / or videos in the learning process, failing to effectively explore other valuable feature areas, thereby limiting its overall discrimination ability. SUMMARY

[0004] The purpose of the present disclosure is to provide a method and apparatus for training of a time-erasing-based neural network capable of improving overall prediction accuracy, improving computational efficiency, and reducing computational overhead.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for training of a time-erasing-based neural network includes: obtaining a training sample including multimedia data; for each time step in a plurality of time steps, generating a first feature map corresponding to the time step by inputting the obtained training sample into a convolution layer; generating a second feature map corresponding to the time step by inputting the first feature map corresponding to the time step into a pooling layer; generating a first prediction map corresponding to the time step by inputting the second feature map corresponding to the time step into a first fully connected layer; generating a loss corresponding to the time step based on the first prediction map corresponding to the time step and a real label; and obtaining a trained time-erasing-based neural network by adjusting parameters of the time-erasing-based neural network based on the loss corresponding to the time step, wherein for each time step except the first time step, the step of generating the second feature map corresponding to the time step by inputting the first feature map corresponding to the time step into the pooling layer includes: applying an erasing mask to the first feature map corresponding to the time step; and generating the second feature map corresponding to the time step by inputting the first feature map to which the erasing mask is applied into the pooling layer, wherein the erasing mask varies with each time step.

[0006] Optionally, the training method further comprises: for each time step except the first time step, generating an average second prediction map corresponding to all previous time steps before the time step; generating a normalized average second prediction map corresponding to all previous time steps before the time step by normalizing the average second prediction map corresponding to all previous time steps before the time step; generating an erasing mask by suppressing regions associated with higher prediction probability scores based on the normalized average second prediction map corresponding to all previous time steps before the time step.

[0007] Optionally, the step of generating the average second prediction map corresponding to all previous time steps before the time step comprises: generating respective second prediction maps corresponding to all previous time steps before the time step by respectively inputting all first feature maps corresponding to all previous time steps before the time step into a second fully connected layer; generating the average second prediction map corresponding to all previous time steps before the time step by averaging the respective second prediction maps corresponding to all previous time steps before the time step.

[0008] Optionally, the step of generating the normalized average second prediction map corresponding to all previous time steps before the time step by normalizing the average second prediction map corresponding to all previous time steps before the time step comprises: generating probability scores of different classes corresponding to all previous time steps before the time step as the normalized average second prediction map corresponding to all previous time steps before the time step by applying a soft-max function to the average second prediction map corresponding to all previous time steps before the time step along a class dimension.

[0009] Optionally, the step of generating the erasing mask by suppressing regions associated with higher prediction probability scores based on the normalized average second prediction map corresponding to all previous time steps before the time step comprises: for each position, generating a first value as a value of the erasing mask at the position when a value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is greater than a threshold value; for each position, generating a second value different from the first value as the value of the erasing mask at the position when the value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is less than the threshold value.

[0010] Optionally, the threshold value comprises a dynamic threshold value and a fixed threshold value, and the dynamic threshold value varies based on the time step.

[0011] Optionally, the training method further comprises: for each position, generating a dynamic threshold corresponding to the position based on a mean and a standard deviation of values of all channels of the normalized average second prediction map at the position corresponding to all previous time steps before the time step.

[0012] Optionally, for each position, the step of generating a dynamic threshold corresponding to the position based on a mean and a standard deviation of values of all channels of the normalized average second prediction map at the position corresponding to all previous time steps before the time step comprises: for each position, summing the mean and a product of the standard deviation and a hyperparameter as the dynamic threshold corresponding to the position.

[0013] Optionally, for the first time step, the step of generating the second feature map corresponding to the time step by inputting the first feature map corresponding to the time step into the pooling layer comprises: generating the second feature map corresponding to the time step by directly inputting the first feature map corresponding to the time step into the pooling layer.

[0014] Optionally, for the first time step, the step of generating the loss corresponding to the time step based on the first prediction map corresponding to the time step comprises: generating the loss corresponding to the time step by applying the first prediction map corresponding to the time step and the ground truth label to a cross-entropy loss function, wherein for each time step other than the first time step, the step of generating the loss corresponding to the time step based on the first prediction map corresponding to the time step comprises: generating a sub-loss corresponding to the time step by applying the first prediction map corresponding to the time step and the ground truth label to a cross-entropy loss function; generating respective sub-losses corresponding to all previous time steps before the time step by applying respective first prediction maps corresponding to all previous time steps before the time step and the ground truth label to respective cross-entropy loss functions; summing the respective sub-losses corresponding to all previous time steps before the time step and the sub-loss corresponding to the time step as the loss corresponding to the time step.

[0015] According to a second aspect of embodiments of the present disclosure, a prediction method based on a time-erasing neural network comprises: obtaining to-be-predicted data comprising multimedia data; generating a first prediction map as a prediction result by inputting the to-be-predicted data into a time-erasing neural network, wherein the time-erasing neural network is obtained by training based on the above-mentioned training method of the time-erasing neural network.

[0016] According to a third aspect of embodiments of the present disclosure, a training device of a time self-erasing neural network comprises: a sample obtaining module configured to obtain a training sample comprising multimedia data; a training module configured to, for each time step of a plurality of time steps, generate a first feature map corresponding to the time step by inputting the obtained training sample into a convolution layer, generate a second feature map corresponding to the time step by inputting the first feature map corresponding to the time step into a pooling layer, and generate a first prediction map corresponding to the time step by inputting the second feature map corresponding to the time step into a first fully connected layer; a loss generating module configured to generate a loss corresponding to the time step based on the first prediction map corresponding to the time step and a real label; and a parameter adjusting module configured to obtain a trained time self-erasing neural network by adjusting parameters of the time self-erasing neural network based on the loss corresponding to the time step, wherein, for each time step except for a first time step, the training module is configured to apply an erasing mask to the first feature map corresponding to the time step, and generate the second feature map corresponding to the time step by inputting the first feature map to which the erasing mask is applied into the pooling layer, wherein the erasing mask varies with each time step.

[0017] Optionally, the training device further comprises an erasing mask generating module configured to, for each time step except for the first time step, generate an average second prediction map corresponding to all previous time steps before the time step, generate a normalized average second prediction map corresponding to all previous time steps before the time step by normalizing the average second prediction map corresponding to all previous time steps before the time step, and generate the erasing mask by suppressing a region associated with a higher prediction probability score based on the normalized average second prediction map corresponding to all previous time steps before the time step.

[0018] Optionally, the erasing mask generating module is configured to generate respective second prediction maps corresponding to all previous time steps before the time step by inputting all first feature maps corresponding to all previous time steps before the time step into a second fully connected layer respectively, and generate the average second prediction map corresponding to all previous time steps before the time step by averaging the respective second prediction maps corresponding to all previous time steps before the time step.

[0019] Optionally, the erasure mask generation module is configured to generate, as the normalized average second prediction map corresponding to all previous time steps before the time step, probability scores of different categories corresponding to all previous time steps before the time step by applying a soft-max function to the average second prediction map corresponding to all previous time steps before the time step along the category dimension.

[0020] Optionally, the erasure mask generation module is configured to, for each position, generate, as a value of the erasure mask at the position, a first value when a value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is greater than a threshold value; and generate, as the value of the erasure mask at the position, a second value different from the first value when the value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is less than the threshold value.

[0021] Optionally, the threshold value comprises a dynamic threshold value and a fixed threshold value, and the dynamic threshold value varies based on the time step.

[0022] Optionally, the training device further comprises a dynamic threshold value generation module configured to, for each position, generate a dynamic threshold value corresponding to the position based on a mean and a standard deviation of values of all channels of the normalized average second prediction map corresponding to all previous time steps before the time step at the position.

[0023] Optionally, the dynamic threshold value generation module is configured to, for each position, sum the mean and a product of the standard deviation and a hyperparameter as the dynamic threshold value corresponding to the position.

[0024] Optionally, for the first time step, the training module is configured to generate the second feature map corresponding to the time step by directly inputting the first feature map corresponding to the time step into the pooling layer.

[0025] Optionally, for the first time step, the loss generation module is configured to generate a loss corresponding to the time step by applying the first prediction map corresponding to the time step and the ground truth label to a cross-entropy loss function, wherein for each time step other than the first time step, the loss generation module is configured to generate a sub-loss corresponding to the time step by applying the first prediction map corresponding to the time step and the ground truth label to a cross-entropy loss function, generate respective sub-losses corresponding to all previous time steps before the time step by applying respective first prediction maps corresponding to the all previous time steps before the time step and the ground truth label to respective cross-entropy loss functions, and take a sum of the respective sub-losses corresponding to the all previous time steps before the time step and the sub-loss corresponding to the time step as the loss corresponding to the time step.

[0026] According to a fourth aspect of embodiments of the present disclosure, a prediction device based on a time self-erasing neural network includes: a data acquisition module configured to acquire to-be-predicted data including multimedia data; and a prediction module configured to generate a first prediction map as a prediction result by inputting the to-be-predicted data into a time self-erasing neural network, wherein the time self-erasing neural network is obtained by training a training device based on a time self-erasing neural network.

[0027] According to a fifth aspect of embodiments of the present disclosure, an electronic device includes at least one processor and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to perform the training method based on a time self-erasing neural network or the prediction method based on a time self-erasing neural network.

[0028] According to a sixth aspect of embodiments of the present disclosure, a computer readable storage medium, when storing instructions stored therein are executed by at least one processor, cause the at least one processor to perform the training method based on a time self-erasing neural network or the prediction method based on a time self-erasing neural network.

[0029] According to a seventh aspect of embodiments of the present disclosure, a computer program product includes computer instructions, when executed by at least one processor, implement the training method based on a time self-erasing neural network or the prediction method based on a time self-erasing neural network.

[0030] The training method and the training device of the neural network based on time self-erase according to the embodiments of the present disclosure introduce a time self-erase mechanism to optimize the attention range of the neural network by erasing redundant areas step by step. This mechanism enables the network to focus on different feature areas at each time step, improves the diversity and learning efficiency of the feature representation, and avoids the redundancy problem of feature representation in the time dimension in traditional neural networks. In addition, the training device of the neural network based on time self-erase according to the embodiments of the present disclosure generates an erase mask by combining a dynamic threshold and a fixed threshold, so that the neural network can adaptively suppress the areas that have been considered important in the previous time step. This adaptive approach improves the flexibility and effectiveness of the erase mask and avoids the limitations of the fixed threshold strategy. Therefore, the training method and the device of the neural network based on time self-erase and the prediction method and the device according to the embodiments of the present disclosure improve the overall prediction accuracy, improve the computational efficiency, and reduce the computational overhead. BRIEF DESCRIPTION OF DRAWINGS

[0031] The above and / or other aspects of the present disclosure will become apparent and more readily appreciated from the following detailed description, taken in conjunction with the accompanying drawings.

[0032] Figure 1 FIG. 1 is a flowchart illustrating a training method of a neural network based on time self-erase according to an embodiment of the present disclosure.

[0033] Figure 2 FIG. 2 is a flowchart illustrating a method of generating a second feature map according to an embodiment of the present disclosure.

[0034] Figure 3 FIG. 3 is a flowchart illustrating a method of generating a loss according to an embodiment of the present disclosure.

[0035] Figure 4 FIG. 4 is a flowchart illustrating a method of generating an erase mask according to an embodiment of the present disclosure.

[0036] Figure 5 FIG. 5 is a flowchart illustrating a method of generating an average second prediction map according to an embodiment of the present disclosure.

[0037] Figure 6 FIG. 6 is a flowchart illustrating a prediction method of a neural network based on time self-erase according to an embodiment of the present disclosure.

[0038] Figure 7 FIG. 7 is a block diagram illustrating a training device of a neural network based on time self-erase according to an embodiment of the present disclosure.

[0039] Figure 8 FIG. 8 is a block diagram illustrating a prediction device of a neural network based on time self-erase according to an embodiment of the present disclosure.

[0040] Figure 9 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0041] Throughout the drawings and detailed description, unless otherwise described or provided, the same drawing reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0042] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for increased clarity and conciseness, descriptions of features known after understanding the disclosure of the present application may be omitted.

[0043] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein that will be apparent after understanding the disclosure of this application.

[0044] Although terms such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions are not limited by these terms. Instead, these terms are used only to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Thus, what is referred to as a first member, first component, first region, first layer, or first portion in the examples described herein may also be referred to as a second member, second component, second region, second layer, or second portion without departing from the teachings of the examples.

[0045] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise," "include," and "have" indicate the presence of the stated features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0046] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as those generally understood by those skilled in the art based on the understanding of the disclosure of this application and the art to which this disclosure belongs. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the disclosure of this application, and should not be interpreted in an idealized or overly formal manner. The use of the term "may" with respect to an example or embodiment herein (e.g., with respect to what an example or embodiment may include or implement) indicates that there is at least one example or embodiment that includes or implements such a feature, and all examples are not limited thereto.

[0047] Unless otherwise specified, the same reference numerals generally refer to the same elements (e.g., components, steps, and methods). Reference numerals described in previous embodiments that appear again in later embodiments may be omitted. Furthermore, technical features described in different or the same embodiments may be combined in any manner, as long as the combined embodiment or technical solution is complete and can solve the technical problems of this application or achieve technical effects described or not described in this disclosure but determinable based on the above complete technical solution. The following describes the terms used in this disclosure.

[0048] It should be noted that, in the absence of conflicts between the various embodiments, these embodiments and their features may be combined with each other.

[0049] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0050] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0051] Figure 1 is a flowchart illustrating a training method of a neural network based on time self-erasing according to an embodiment of the present disclosure.

[0052] Reference Figure 1 In step S110, a training sample including multimedia data is obtained. According to an embodiment of the present disclosure, the multimedia data may include at least one of image data, video data, etc., but is not limited thereto.

[0053] At step S120, for each of the plurality of time steps, a first feature map corresponding to the time step is generated by inputting the obtained training sample into a convolutional layer. According to an embodiment of the present disclosure, the time self-erasing based neural network can be a spiking neural network (SNN), but the training method of the time self-erasing based neural network according to the present disclosure is not limited thereto. According to an embodiment of the present disclosure, the time self-erasing based neural network can be used to implement a classification, regression and / or object detection task, but is not limited thereto. According to an embodiment of the present disclosure, the time self-erasing based neural network can include a convolutional layer, an input of the convolutional layer can be connected to an input of the time self-erasing based neural network, and the first feature map can be an output of the convolutional layer.

[0054] At step S130, a second feature map corresponding to the time step is generated by inputting the first feature map corresponding to the time step into a pooling layer. According to an embodiment of the present disclosure, the time self-erasing based neural network can further include a pooling layer, the pooling layer can be a global average pooling layer, an input of the pooling layer can be connected to an output of the convolutional layer, and the second feature map can be an output of the pooling layer. In the following, step S130 will be described in detail with reference to Figure 2 Step S130 is described in detail.

[0055] At step S140, a first prediction map corresponding to the time step is generated by inputting the second feature map corresponding to the time step into a first fully connected layer. According to an embodiment of the present disclosure, the time self-erasing based neural network can further include a first fully connected layer, an input of the first fully connected layer can be connected to an output of the pooling layer, and the first prediction map can be an output of the first fully connected layer.

[0056] At step S150, a loss corresponding to the time step is generated based on the first prediction map corresponding to the time step and a real label. In the following, step S150 will be described in detail with reference to Figure 3 Step S150 is described in detail.

[0057] At step S160, a trained time self-erasing based neural network is obtained by adjusting parameters of the time self-erasing based neural network based on the loss corresponding to the time step.

[0058] Figure 2 is a flow chart illustrating a method of generating a second feature map according to an embodiment of the present disclosure.

[0059] Reference is made to Figure 2In step S131, for each time step except the first time step, an erasure mask is applied to the first feature map corresponding to the time step. In step S132, a second feature map corresponding to the time step is generated by inputting the first feature map to which the erasure mask is applied into a pooling layer. According to an embodiment of the present disclosure, the erasure mask may vary with each time step. Alternatively, in step S133, for the first time step, a second feature map corresponding to the time step is generated by directly inputting the first feature map corresponding to the time step into the pooling layer.

[0060] In the previous time step, the predicted high probability score areas correspond to semantically more discriminative areas. Therefore, in the current time step, we can focus more on other areas by applying an erasure mask before feeding the first feature map into the pooling layer, so as to mine more potential informative areas.

[0061] Figure 3 is a flowchart illustrating a method of generating losses according to an embodiment of the present disclosure.

[0062] Reference Figure 3 In step S151, for the first time step, a loss corresponding to the time step is generated by applying the first predicted image corresponding to the time step and the true label to the cross entropy loss function. In one example, step S151 can be shown as the following formula 1, where, represents the loss function, represents the cross entropy loss function, represents the prediction result of the first time step, represents the true label.

[0063] Formula 1:

[0064]

[0065] In step S152, for each time step except the first time step, a sub-loss corresponding to the time step is generated by applying the first prediction map and the true label corresponding to the time step to the cross entropy loss function. In step S153, each sub-loss corresponding to all previous time steps before the time step is generated by applying the respective first prediction maps and the true labels corresponding to all previous time steps before the time step to the cross entropy loss function. In step S154, the sum of the respective sub-losses corresponding to all previous time steps before the time step and the sub-loss corresponding to the time step is used as the loss corresponding to the time step. In one example, steps S152 to S154 can be shown as the following formula 2, wherein, represents the time step, denotes a current time step, denotes the first prediction map generated in step S140 (i.e., the first prediction map generated by sequentially inputting the first feature map to which the erasing mask is applied into the pooling layer and the first fully connected layer).

[0066] Equation 2:

[0067]

[0068] Figure 4 is a flowchart illustrating a method of generating an erasing mask according to an embodiment of the disclosure.

[0069] Referring to Figure 4 In step S170, for each time step other than the first time step, an average second prediction map corresponding to all previous time steps before the time step is generated. In the following, referring to Figure 5 Step S170 is described in detail.

[0070] Figure 5 is a flowchart illustrating a method of generating an average second prediction map according to an embodiment of the disclosure.

[0071] Referring to Figure 5 In step S171, by respectively inputting all first feature maps corresponding to all previous time steps before the time step into the second fully connected layer, respective second prediction maps corresponding to all previous time steps before the time step are generated. According to an embodiment of the disclosure, the second prediction map is distinguished from the first prediction map at least in that, for each time step other than the first time step, the second prediction map is not based on the erasing mask, while the first prediction map is based on the erasing mask.

[0072] In step S172, by averaging the respective second prediction maps corresponding to all previous time steps before the time step, an average second prediction map corresponding to all previous time steps before the time step is generated.

[0073] Referring back to Figure 4 In step S180, by normalizing the average second prediction map corresponding to all previous time steps before the time step, a normalized average second prediction map corresponding to all previous time steps before the time step is generated.

[0074] In one example, as shown in Equation 3 below, probability scores for different classes corresponding to all previous time steps before the time step can be generated as normalized average second prediction maps corresponding to all previous time steps before the time step by applying a Softmax function along the class dimension to the average second prediction maps corresponding to all previous time steps before the time step, where, denotes normalized average second prediction maps corresponding to all previous time steps before the time step, denotes an index corresponding to a time step, denotes a second prediction map.

[0075] Equation 3:

[0076]

[0077] At step S190, an erasing mask is generated by suppressing regions associated with higher prediction probability scores based on the normalized average second prediction maps corresponding to all previous time steps before the time step.

[0078] According to an embodiment of the present disclosure, for each position, a first value is generated as a value of the erasing mask at the position when a value of the normalized average second prediction maps corresponding to all previous time steps before the time step at the position is greater than a threshold value. In one example, the first value can be 0. In addition, for each position, a second value different from the first value is generated as a value of the erasing mask at the position when the value of the normalized average second prediction maps corresponding to all previous time steps before the time step at the position is less than the threshold value. In one example, the second value can be 1.

[0079] According to an embodiment of the present disclosure, the threshold value includes a dynamic threshold value and a fixed threshold value, and the dynamic threshold value varies based on the time step. In one example, the erasing mask can be calculated as shown in Equation 4 below, where, denotes a mask value for each position (i, j), denotes a value of the normalized average second prediction maps at each position (i, j), denotes a fixed threshold value, denotes a dynamic threshold value, 0 indicates to suppress the region, and 1 indicates to keep the region.

[0080] Equation 4:

[0081] According to an embodiment of the present disclosure, for each position, a dynamic threshold value corresponding to the position can be generated based on a mean and a standard deviation of values of all channels of the normalized average second prediction map at the position corresponding to all previous time steps before the time step.

[0082] In one example, for each position, a product of the standard deviation and a hyperparameter and a sum of the mean can be taken as the dynamic threshold value corresponding to the position, as shown in Equation 5 below, where, denotes the mean of the normalized average second prediction map, denotes the standard deviation of the normalized average second prediction map, denotes the hyperparameter.

[0083] Equation 5:

[0084] The training method of the time self-erasing based neural network according to an embodiment of the present disclosure introduces a time self-erasing mechanism to optimize the attention range of the neural network by gradually erasing redundant regions. This mechanism enables the network to focus on different feature regions at each time step, improving the diversity and learning efficiency of feature representation and avoiding the redundancy of feature representation in the time dimension in traditional neural networks. In addition, the training method of the time self-erasing based neural network according to an embodiment of the present disclosure generates an erasing mask by combining dynamic threshold values and fixed threshold values, enabling the neural network to adaptively suppress regions that have been considered important in the previous time step. This adaptive approach improves the flexibility and effectiveness of the erasing mask, avoiding the limitations of the fixed threshold value strategy. Therefore, the training method of the time self-erasing based neural network according to an embodiment of the present disclosure improves the overall prediction accuracy, improves the computational efficiency, and reduces the computational overhead.

[0085] Figure 6 FIG. 8 is a flowchart illustrating a prediction method of a time self-erasing based neural network according to an embodiment of the present disclosure.

[0086] In step S210, the to-be-predicted data including multimedia data is obtained. In step S220, a first prediction map is generated as a prediction result by inputting the to-be-predicted data into the time self-erasing based neural network. According to an embodiment of the present disclosure, the time self-erasing based neural network is obtained by training according to the above-mentioned training method of the time self-erasing based neural network, and therefore the prediction method of the time self-erasing based neural network according to an embodiment of the present disclosure improves the overall prediction accuracy, improves the computational efficiency, and reduces the computational overhead.

[0087] Figure 7 FIG. 9 is a block diagram illustrating a training device of a time self-erasing based neural network according to an embodiment of the present disclosure.

[0088] Reference Figure 7 , the training device 100 may include a sample acquisition module 110 , a training module 120 , a loss generation module 130 and a parameter adjustment module 140 .

[0089] The sample acquisition module 110 is configured to acquire training samples including multimedia data.

[0090] The training module 120 is configured to: for each time step in a plurality of time steps, generate a first feature map corresponding to the time step by inputting the acquired training sample into the convolution layer; generate a second feature map corresponding to the time step by inputting the first feature map corresponding to the time step into the pooling layer; and generate a first prediction map corresponding to the time step by inputting the second feature map corresponding to the time step into the first fully connected layer.

[0091] According to an embodiment of the present disclosure, for each time step except the first time step, the training module 120 is configured to: apply an erasure mask to the first feature map corresponding to the time step. The erasure mask changes with each time step; and generate a second feature map corresponding to the time step by inputting the first feature map to which the erasure mask is applied into a pooling layer. For the first time step, the training module 120 is configured to: generate a second feature map corresponding to the time step by directly inputting the first feature map corresponding to the time step into the pooling layer.

[0092] The loss generation module 130 is configured to generate a loss corresponding to the time step based on the first prediction map corresponding to the time step and the true label.

[0093] According to an embodiment of the present disclosure, for the first time step, the loss generation module 130 is configured to: generate a loss corresponding to the time step by applying the first prediction map and the true label corresponding to the time step to the cross entropy loss function. In addition, for each time step other than the first time step, the loss generation module 130 is configured to: generate a sub-loss corresponding to the time step by applying the first prediction map and the true label corresponding to the time step to the cross entropy loss function; generate each sub-loss corresponding to all previous time steps before the time step by applying each first prediction map and the true label corresponding to all previous time steps before the time step to the cross entropy loss function respectively; and use the sum of each sub-loss corresponding to all previous time steps before the time step and the sub-loss corresponding to the time step as the loss corresponding to the time step.

[0094] The parameter adjustment module 140 is configured to obtain the trained time self-erasing based neural network by adjusting parameters of the time self-erasing based neural network based on the loss corresponding to the time step.

[0095] According to an embodiment of the present disclosure, the training device 100 can further include an erasing mask generation module 150. The erasing mask generation module 150 is configured to, for each time step except the first time step, generate an average second prediction map corresponding to all previous time steps before the time step.

[0096] According to an embodiment of the present disclosure, the erasing mask generation module 150 is configured to generate, by respectively inputting all first feature maps corresponding to all previous time steps before the time step into a second fully connected layer, respective second prediction maps corresponding to all previous time steps before the time step; and generate, by averaging the respective second prediction maps corresponding to all previous time steps before the time step, an average second prediction map corresponding to all previous time steps before the time step.

[0097] According to an embodiment of the present disclosure, the erasing mask generation module 150 is configured to generate, by normalizing the average second prediction map corresponding to all previous time steps before the time step, a normalized average second prediction map corresponding to all previous time steps before the time step.

[0098] According to an embodiment of the present disclosure, the erasing mask generation module 150 is configured to generate, by applying a soft-max function to the average second prediction map corresponding to all previous time steps before the time step along a category dimension, probability scores of different categories corresponding to all previous time steps before the time step as the normalized average second prediction map corresponding to all previous time steps before the time step.

[0099] According to an embodiment of the present disclosure, the erasing mask generation module 150 is configured to generate, by suppressing regions related to higher prediction probability scores based on the normalized average second prediction map corresponding to all previous time steps before the time step, an erasing mask.

[0100] According to an embodiment of the present disclosure, the erasing mask generation module 150 is configured to, for each position, generate, when a value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is greater than a threshold value, a first value as a value of the erasing mask at the position; and for each position, generate, when the value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is less than the threshold value, a second value different from the first value as the value of the erasing mask at the position.

[0101] According to an embodiment of the present disclosure, the threshold value comprises a dynamic threshold value and a fixed threshold value, and the dynamic threshold value varies based on the time step. According to an embodiment of the present disclosure, the training device 100 can further comprise a dynamic threshold value generation module 160. The dynamic threshold value generation module 160 is configured to, for each position, generate a dynamic threshold value corresponding to the position based on a mean and a standard deviation of values of all channels of the normalized average second prediction map at the position corresponding to all previous time steps before the time step.

[0102] According to an embodiment of the present disclosure, the dynamic threshold value generation module 160 is configured to, for each position, generate the dynamic threshold value corresponding to the position as a sum of the mean and a product of the standard deviation and a hyperparameter.

[0103] The training device of the neural network based on time self-erasing according to an embodiment of the present disclosure introduces a time self-erasing mechanism to optimize the attention range of the neural network by gradually erasing redundant regions. This mechanism enables the network to focus on different feature regions at each time step, improving the diversity and learning efficiency of feature representation and avoiding the redundancy problem of feature representation in the time dimension in traditional neural networks. In addition, the training device of the neural network based on time self-erasing according to an embodiment of the present disclosure generates the erasing mask by combining the dynamic threshold value and the fixed threshold value, so that the neural network can adaptively suppress the regions that have been considered important in the previous time step. This adaptive approach improves the flexibility and effectiveness of the erasing mask, avoiding the limitations of the fixed threshold value strategy. Therefore, the training device of the neural network based on time self-erasing according to an embodiment of the present disclosure improves the overall prediction accuracy, improves the computational efficiency, and reduces the computational overhead.

[0104] Figure 8 is a block diagram illustrating a prediction device of a neural network based on time self-erasing according to an embodiment of the present disclosure.

[0105] With reference to Figure 8 , the prediction device 200 comprises a data acquisition module 210 and a prediction module 220.

[0106] The data acquisition module 210 is configured to acquire to-be-predicted data comprising multimedia data. The prediction module 220 is configured to generate a first prediction map as a prediction result by inputting the to-be-predicted data into the neural network based on time self-erasing. According to an embodiment of the present disclosure, the neural network based on time self-erasing is obtained by training the training device 100 of the neural network based on time self-erasing according to the above, and therefore the prediction device of the neural network based on time self-erasing according to an embodiment of the present disclosure improves the overall prediction accuracy, improves the computational efficiency, and reduces the computational overhead.

[0107] Figure 9 is a block diagram illustrating an electronic device according to an embodiment of the disclosure.

[0108] Referring to Figure 9 , the electronic device 300 includes at least one processor 310 and at least one memory 320 storing computer-executable instructions that, when executed by the at least one processor 310, cause the at least one processor 310 to perform a training method and / or a prediction method of a time-autodeleting neural network according to an embodiment of the disclosure.

[0109] In the electronic device 300, the processor 310 can include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor can also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0110] The processor 310 can execute instructions or code stored in the memory 320, where the memory 320 can also store data. The instructions and data can also be transmitted and received over a network via a network interface device, which can employ any known transmission protocol.

[0111] The memory 320 can be integral with the processor 310, e.g., RAM or flash memory disposed within an integrated circuit microprocessor, etc. Further, the memory 320 can include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 320 and the processor 310 can be operatively coupled or can communicate with each other, e.g., through I / O ports, network connections, etc., such that the processor 310 can read files stored in the memory.

[0112] Optionally, the electronic device 300 can also include a video display, such as a liquid crystal display, and a user interaction interface, such as a keyboard, a mouse, a touch input device, etc. All components of the electronic device 300 can be connected to each other via a bus and / or a network.

[0113] According to embodiments of the present disclosure, a computer-readable storage medium can also be provided, in which instructions stored in the computer-readable storage medium, when executed by at least one processor, cause the at least one processor to perform the above-described training method of a time self-erasing neural network or the above-described prediction method of a time self-erasing neural network. Examples of the computer-readable storage medium herein include a read-only memory (ROM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a BD-R, a BD-R LTH, a BD-RE, a Blu-ray or an optical disc memory, a hard disk drive (HDD), a solid state drive (SSD), a card memory (such as a multimedia card, a secure digital (SD) card, or an extreme digital (XD) card), a magnetic tape, a floppy disk, a magneto-optical data storage device, an optical data storage device, a hard disk, a solid state disk, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and provide the computer program and any associated data, data files, and data structures to a processor or a computer so that the processor or the computer can execute the computer program. The computer program in the above-described computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc., and in addition, in one example, the computer program and any associated data, data files, and data structures are distributed over a networked computer system so that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0114] According to embodiments of the present disclosure, a computer program product can also be provided, including computer instructions that, when executed by at least one processor, implement the above-described training method of a time self-erasing neural network or the above-described prediction method of a time self-erasing neural network.

[0115] According to the embodiment of the present disclosure, the training method and training device of the neural network based on time self-erasing introduce a time self-erasing mechanism to optimize the attention range of the neural network by gradually erasing redundant areas. This mechanism enables the network to focus on different feature areas at each time step, improves the diversity and learning efficiency of feature representation, and avoids the redundancy problem of feature representation in the time dimension in traditional neural networks. In addition, according to the embodiment of the present disclosure, the training device of the neural network based on time self-erasing generates an erasure mask by combining a dynamic threshold with a fixed threshold, so that the neural network can adaptively suppress areas that have been considered important in the previous time step. This adaptive approach improves the flexibility and effectiveness of the erasure mask and avoids the limitations of the fixed threshold strategy. Therefore, the training method and device of the neural network based on time self-erasing and the prediction method and device according to the embodiment of the present disclosure improve the overall prediction accuracy, improve the computational efficiency, and reduce the computational overhead.

[0116] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0117] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for a time-based self-erasing neural network, wherein the time-based self-erasing neural network is used for image classification, characterized in that: The training method comprises: obtaining a training sample comprising image data; for each time step of a plurality of time steps, generating a first feature map corresponding to the time step and the image data by inputting the obtained training sample comprising the image data into a convolutional layer; generating a second feature map corresponding to the time step and the image data by inputting the first feature map corresponding to the time step and the image data into a pooling layer; generating a first prediction map corresponding to the time step and the image data by inputting the second feature map corresponding to the time step and the image data into a first fully connected layer; generating a loss corresponding to the time step and the image data based on the first prediction map corresponding to the time step and the image data and a true label; obtaining a trained time self-erasing neural network for image classification by adjusting parameters of the time self-erasing neural network based on the loss corresponding to the time step and the image data, wherein for each time step other than the first time step, the step of generating a second feature map corresponding to the time step and the image data by inputting the first feature map corresponding to the time step and the image data into a pooling layer comprises: applying an erasure mask to the first feature map corresponding to the time step and the image data; generating a second feature map corresponding to the time step and the image data by inputting the first feature map to which the erasure mask is applied into a pooling layer, wherein the erasure mask varies with each time step.

2. The training method of claim 1, wherein, The training method further comprises: for each time step other than the first time step, generating an average second prediction map corresponding to all previous time steps before the time step and the image data; generating a normalized average second prediction map corresponding to all previous time steps before the time step and the image data by normalizing the average second prediction map corresponding to all previous time steps before the time step and the image data; generating an erasure mask by suppressing regions associated with higher prediction probability scores based on the normalized average second prediction map corresponding to all previous time steps before the time step and the image data.

3. The training method of claim 2, wherein, The step of generating an average second prediction map corresponding to all previous time steps before the time step and the image data comprises: generating respective second prediction maps corresponding to all previous time steps before the time step and the image data by inputting all first feature maps corresponding to all previous time steps before the time step and the image data into respective second fully connected layers; generating an average second prediction map corresponding to all previous time steps before the time step and the image data by averaging the respective second prediction maps corresponding to all previous time steps before the time step and the image data.

4. The training method of claim 2, wherein, The step of generating a normalized average second prediction map corresponding to all previous time steps preceding the time step and the image data by normalizing the average second prediction map corresponding to all previous time steps preceding the time step and the image data comprises: The step of generating probability scores for different classes corresponding to the normalized average second prediction map corresponding to all previous time steps preceding the time step and the image data as the normalized average second prediction map corresponding to all previous time steps preceding the time step and the image data by applying a soft-max function to the average second prediction map corresponding to all previous time steps preceding the time step and the image data along the class dimension.

5. The training method of claim 2, wherein, The step of generating an erasing mask by suppressing regions associated with higher prediction probability scores based on the normalized average second prediction map corresponding to all previous time steps preceding the time step and the image data comprises: For each location, generating a first value as a value of the erasing mask at the location when a value of the normalized average second prediction map corresponding to all previous time steps preceding the time step and the image data at the location is greater than a threshold value; For each location, generating a second value different from the first value as a value of the erasing mask at the location when the value of the normalized average second prediction map corresponding to all previous time steps preceding the time step and the image data at the location is less than the threshold value.

6. The training method of claim 1, wherein, For the first time step, the step of generating a second feature map corresponding to the time step and the image data by inputting a first feature map corresponding to the time step and the image data to a pooling layer comprises: The step of generating a second feature map corresponding to the time step and the image data by directly inputting a first feature map corresponding to the time step and the image data to a pooling layer.

7. The training method of claim 1, wherein, For the first time step, the step of generating a loss corresponding to the time step and the image data based on a first prediction map corresponding to the time step and the image data comprises: The step of generating a loss corresponding to the time step and the image data by applying the first prediction map corresponding to the time step and the image data and a ground truth label to a cross-entropy loss function, For each time step other than the first time step and the image data, the step of generating a loss corresponding to the time step and the image data based on a first prediction map corresponding to the time step and the image data comprises: The step of generating a sub-loss corresponding to the time step and the image data by applying the first prediction map corresponding to the time step and the image data and a ground truth label to a cross-entropy loss function; The step of generating respective sub-losses corresponding to all previous time steps preceding the time step and the image data by applying respective first prediction maps corresponding to all previous time steps preceding the time step and the image data and respective ground truth labels to respective cross-entropy loss functions; summing each sub-loss corresponding to all previous time steps before the time step and the image data and the sub-loss corresponding to the time step and the image data as a loss corresponding to the time step and the image data.

8. A training device of a time-autocanceling neural network for image classification, the time-autocanceling neural network comprising: The training device comprises: a sample obtaining module configured to obtain a training sample comprising image data; a training module configured to, for each time step of a plurality of time steps, generate a first feature map corresponding to the time step and the image data by inputting the obtained training sample into a convolution layer, generate a second feature map corresponding to the time step and the image data by inputting the first feature map corresponding to the time step and the image data into a pooling layer, and generate a first prediction map corresponding to the time step and the image data by inputting the second feature map corresponding to the time step and the image data into a first fully connected layer; a loss generating module configured to generate a loss corresponding to the time step and the image data based on the first prediction map corresponding to the time step and the image data and a real label; a parameter adjusting module configured to obtain a trained time self-erasing based neural network for image classification by adjusting parameters of the time self-erasing based neural network based on the loss corresponding to the time step and the image data, wherein, for each time step except the first time step, the training module is configured to: apply an erasing mask to the first feature map corresponding to the time step and the image data; generate the second feature map corresponding to the time step and the image data by inputting the first feature map to which the erasing mask is applied into the pooling layer, wherein the erasing mask varies with each time step.

9. A computer-readable storage medium, characterized in that, The computer program product comprises a computer readable storage medium having program instructions stored in the computer readable storage medium, the program instructions executable by at least one processor for causing the at least one processor to perform the training method of the time self-erasing based neural network according to any one of claims 1 to 7 when the program instructions stored in the computer readable storage medium are executed by the at least one processor.

10. A computer program product comprising computer instructions, characterized in that, The computer program product comprises a computer readable storage medium having program instructions stored in the computer readable storage medium, the program instructions executable by at least one processor for causing the at least one processor to perform the training method of the time self-erasing based neural network according to any one of claims 1 to 7 when the program instructions stored in the computer readable storage medium are executed by the at least one processor.

Images