Neural network training method and device based on time self-erasing

Abstract

The invention provides a neural network training method and device based on time self-erasing. The training method comprises the following steps: for each time step length, inputting a training sample into a convolutional layer to generate a first feature map corresponding to the time step length; for each time step other than the first time step, generating a second feature map by inputting the first feature map to which an erase mask that varies with each time step is applied into the pooling layer; generating a first prediction map by inputting the second feature map into the first full connection layer; generating a loss based on the first prediction map and the real tag; and adjusting the parameters of the neural network based on the loss to obtain a trained neural network. Therefore, the overall prediction accuracy is improved, the calculation efficiency is improved, and the calculation overhead is reduced.

Description

Background Art

[0002] In the existing technology, spiking neural networks (SNNs), as biologically plausible neural network models, have attracted widespread attention in recent years. Compared with traditional artificial neural networks (ANNs), SNNs transmit information in multiple time steps through pulse signals and can simulate the dynamic process of biological neurons.

[0003] However, despite the advantages of dynamic reasoning, SNNs still have bottlenecks in feature discrimination capabilities, mainly because traditional training methods usually produce the same back-propagation gradients and lack diversity in feature representation. This problem causes SNNs to focus on the same areas of multimedia data such as images and / or videos during learning, failing to effectively explore other valuable feature areas, thus limiting their overall discrimination capabilities. Summary of the invention

[0004] The purpose of the present disclosure is to provide a training method and device for a time-self-erasing neural network that can improve overall prediction accuracy, improve computational efficiency, and reduce computational overhead.

[0005] According to a first aspect of an embodiment of the present disclosure, a training method for a neural network based on time self-erasing includes: acquiring a training sample including multimedia data; for each time step in a plurality of time steps, inputting the acquired training sample into a convolutional layer to generate a first feature map corresponding to the time step; inputting the first feature map corresponding to the time step into a pooling layer to generate a second feature map corresponding to the time step; inputting the second feature map corresponding to the time step into a first fully connected layer to generate a first prediction map corresponding to the time step; based on the first prediction map corresponding to the time step and a true label, generating a first feature map corresponding to the time step; The loss corresponding to the time step; adjusting the parameters of the time self-erasing neural network based on the loss corresponding to the time step to obtain a trained time self-erasing neural network, wherein, for each time step except the first time step, by inputting the first feature map corresponding to the time step into the pooling layer, the step of generating the second feature map corresponding to the time step includes: applying an erasure mask to the first feature map corresponding to the time step; generating the second feature map corresponding to the time step by inputting the first feature map to which the erasure mask is applied into the pooling layer, wherein the erasure mask changes with each time step.

[0006] Optionally, the training method also includes: for each time step except the first time step, generating an average second prediction map corresponding to all previous time steps before the time step; generating a normalized average second prediction map corresponding to all previous time steps before the time step by normalizing the average second prediction map corresponding to all previous time steps before the time step; and generating an erasure mask by suppressing areas associated with higher prediction probability scores based on the normalized average second prediction map corresponding to all previous time steps before the time step.

[0007] Optionally, the step of generating an average second prediction map corresponding to all previous time steps before the time step includes: generating individual second prediction maps corresponding to all previous time steps before the time step by inputting all first feature maps corresponding to all previous time steps before the time step into a second fully connected layer respectively; and generating an average second prediction map corresponding to all previous time steps before the time step by averaging the individual second prediction maps corresponding to all previous time steps before the time step.

[0008] Optionally, the step of generating a normalized average second prediction map corresponding to all previous time steps before the time step by normalizing the average second prediction map corresponding to all previous time steps before the time step includes: generating probability scores of different categories corresponding to all previous time steps before the time step as the normalized average second prediction map corresponding to all previous time steps before the time step by applying a flexible maximum function along the category dimension to the average second prediction map corresponding to all previous time steps before the time step.

[0009] Optionally, the step of generating an erasure mask by suppressing areas associated with higher prediction probability scores based on a normalized average second prediction map corresponding to all previous time steps before the time step includes: for each position, when the value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is greater than a threshold, generating a first value as the value of the erasure mask at the position; for each position, when the value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is less than the threshold, generating a second value different from the first value as the value of the erasure mask at the position.

[0010] Optionally, the threshold comprises a dynamic threshold and a fixed threshold, and the dynamic threshold varies based on the time step.

[0011] Optionally, the training method also includes: for each position, generating a dynamic threshold corresponding to the position based on the mean and standard deviation of the values ​​of all channels of the normalized average second prediction map corresponding to all previous time steps before the time step at the position.

[0012] Optionally, for each position, based on the mean and standard deviation of the values ​​of all channels of the normalized average second prediction map corresponding to all previous time steps before the time step at the position, the step of generating a dynamic threshold corresponding to the position includes: for each position, taking the product of the standard deviation and the hyperparameter and the sum of the mean as the dynamic threshold corresponding to the position.

[0013] Optionally, for the first time step, the step of generating a second feature map corresponding to the time step by inputting the first feature map corresponding to the time step into the pooling layer includes: generating a second feature map corresponding to the time step by directly inputting the first feature map corresponding to the time step into the pooling layer.

[0014] Optionally, for the first time step, based on the first prediction map corresponding to the time step, the step of generating a loss corresponding to the time step includes: generating the loss corresponding to the time step by applying the first prediction map corresponding to the time step and the true label to a cross entropy loss function, wherein, for each time step except the first time step, based on the first prediction map corresponding to the time step, the step of generating a loss corresponding to the time step includes: generating a sub-loss corresponding to the time step by applying the first prediction map corresponding to the time step and the true label to the cross entropy loss function; generating respective sub-losses corresponding to all previous time steps before the time step by using respective first prediction maps and true labels corresponding to all previous time steps before the time step for the cross entropy loss function respectively; and taking the sum of the respective sub-losses corresponding to all previous time steps before the time step and the sub-loss corresponding to the time step as the loss corresponding to the time step.

[0015] According to a second aspect of an embodiment of the present disclosure, a prediction method based on a time self-erasing neural network includes: obtaining data to be predicted including multimedia data; generating a first prediction graph as a prediction result by inputting the data to be predicted into a time self-erasing neural network, wherein the time self-erasing neural network is trained by the above-mentioned time self-erasing neural network training method.

[0016] According to a third aspect of an embodiment of the present disclosure, a training device for a neural network based on time self-erasing includes: a sample acquisition module, configured to: acquire training samples including multimedia data; a training module, configured to: for each time step in a plurality of time steps, generate a first feature map corresponding to the time step by inputting the acquired training samples into a convolutional layer; generate a second feature map corresponding to the time step by inputting the first feature map corresponding to the time step into a pooling layer; generate a first prediction map corresponding to the time step by inputting the second feature map corresponding to the time step into a first fully connected layer; a loss generation module, configured to: The invention is configured as follows: based on the first prediction map corresponding to the time step and the true label, a loss corresponding to the time step is generated; a parameter adjustment module is configured to: adjust the parameters of the time self-erasing neural network based on the loss corresponding to the time step to obtain a trained time self-erasing neural network, wherein, for each time step except the first time step, the training module is configured to: apply an erasure mask to the first feature map corresponding to the time step; and generate a second feature map corresponding to the time step by inputting the first feature map to which the erasure mask is applied into the pooling layer, wherein the erasure mask changes with each time step.

[0017] Optionally, the training device also includes an erasure mask generation module, which is configured to: for each time step except the first time step, generate an average second prediction map corresponding to all previous time steps before the time step; generate a normalized average second prediction map corresponding to all previous time steps before the time step by normalizing the average second prediction map corresponding to all previous time steps before the time step; and generate an erasure mask by suppressing areas associated with higher prediction probability scores based on the normalized average second prediction map corresponding to all previous time steps before the time step.

[0018] Optionally, the erasure mask generation module is configured to: generate respective second prediction maps corresponding to all previous time steps before the time step by inputting all first feature maps corresponding to all previous time steps before the time step into the second fully connected layer respectively; and generate an average second prediction map corresponding to all previous time steps before the time step by averaging the respective second prediction maps corresponding to all previous time steps before the time step.

[0019] Optionally, the erasure mask generation module is configured to generate probability scores for different categories corresponding to all previous time steps before the time step as a normalized average second prediction map corresponding to all previous time steps before the time step by applying a flexible maximum function along the category dimension to the average second prediction map corresponding to all previous time steps before the time step.

[0020] Optionally, the erasure mask generation module is configured to: for each position, when the value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is greater than a threshold, generate a first value as the value of the erasure mask at the position; for each position, when the value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is less than the threshold, generate a second value different from the first value as the value of the erasure mask at the position.

[0021] Optionally, the threshold comprises a dynamic threshold and a fixed threshold, and the dynamic threshold varies based on the time step.

[0022] Optionally, the training device also includes a dynamic threshold generation module, which is configured to: for each position, generate a dynamic threshold corresponding to the position based on the mean and standard deviation of the values ​​of all channels of the normalized average second prediction map corresponding to all previous time steps before the time step at the position.

[0023] Optionally, the dynamic threshold generation module is configured to: for each position, take the product of the standard deviation and the hyperparameter and the sum of the mean as the dynamic threshold corresponding to the position.

[0024] Optionally, for the first time step, the training module is configured to generate a second feature map corresponding to the time step by directly inputting the first feature map corresponding to the time step into the pooling layer.

[0025] Optionally, for the first time step, the loss generation module is configured to: generate a loss corresponding to the time step by applying the first prediction map and the true label corresponding to the time step to a cross entropy loss function, wherein, for each time step except the first time step, the loss generation module is configured to: generate a sub-loss corresponding to the time step by applying the first prediction map and the true label corresponding to the time step to the cross entropy loss function; generate respective sub-losses corresponding to all previous time steps before the time step by using the respective first prediction maps and the true labels corresponding to all previous time steps before the time step for the cross entropy loss function respectively; and take the sum of the respective sub-losses corresponding to all previous time steps before the time step and the sub-loss corresponding to the time step as the loss corresponding to the time step.

[0026] According to a fourth aspect of an embodiment of the present disclosure, a prediction device based on a time self-erasing neural network includes: a data acquisition module, configured to: acquire data to be predicted including multimedia data; a prediction module, configured to: generate a first prediction graph as a prediction result by inputting the data to be predicted into the time self-erasing neural network, wherein the time self-erasing neural network is obtained by training with the above-mentioned time self-erasing neural network training device.

[0027] According to a fifth aspect of an embodiment of the present disclosure, an electronic device includes: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to execute the above-mentioned training method of a time-self-erasing neural network or the above-mentioned prediction method of a time-self-erasing neural network.

[0028] According to the sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium causes at least one processor to execute the above-mentioned training method of a time-self-erasing neural network or the above-mentioned prediction method of a time-self-erasing neural network when instructions stored in the computer-readable storage medium are executed by the at least one processor.

[0029] According to a seventh aspect of an embodiment of the present disclosure, a computer program product includes computer instructions, which, when executed by at least one processor, implement the above-mentioned training method of a time-self-erasing neural network or the above-mentioned prediction method of a time-self-erasing neural network.

[0030] According to the training method and training device of the neural network based on time self-erasing according to the embodiment of the present disclosure, a time self-erasing mechanism is introduced to optimize the attention range of the neural network by gradually erasing redundant areas. This mechanism enables the network to focus on different feature areas at each time step, improves the diversity and learning efficiency of feature representation, and avoids the redundancy problem of feature representation in the time dimension in traditional neural networks. In addition, according to the training device of the neural network based on time self-erasing according to the embodiment of the present disclosure, an erasure mask is generated by combining a dynamic threshold with a fixed threshold, so that the neural network can adaptively suppress the area that has been considered important in the previous time step. This adaptive method improves the flexibility and effectiveness of the erasure mask and avoids the limitations of the fixed threshold strategy. Therefore, according to the training method and device of the neural network based on time self-erasing according to the embodiment of the present disclosure, as well as the prediction method and device, the overall prediction accuracy is improved, the computational efficiency is improved, and the computational overhead is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and / or other aspects of the present disclosure will become clearer and more easily understood through the following detailed description in conjunction with the accompanying drawings.

[0032] Figure 1 is a flowchart illustrating a training method of a neural network based on time self-erasing according to an embodiment of the present disclosure.

[0033] Figure 2 is a flowchart illustrating a method of generating a second feature map according to an embodiment of the present disclosure.

[0034] Figure 3 is a flow chart illustrating a method of generating losses according to an embodiment of the present disclosure.

[0035] Figure 4 is a flowchart illustrating a method of generating an erasure mask according to an embodiment of the present disclosure.

[0036] Figure 5 is a flowchart illustrating a method of generating an average second prediction map according to an embodiment of the present disclosure.

[0037] Figure 6 is a flowchart illustrating a prediction method based on a time self-erasing neural network according to an embodiment of the present disclosure.

[0038] Figure 7 is a block diagram illustrating a training device for a time-self-erasing neural network according to an embodiment of the present disclosure.

[0039] Figure 8 is a block diagram illustrating a prediction device based on a time self-erasing neural network according to an embodiment of the present disclosure.

[0040] Fig. 9 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0041] Throughout the drawings and detailed description, unless otherwise described or provided, the same figure reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0042] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the method, device and / or system described herein. However, after understanding the disclosure of the application, various changes, modifications and equivalents of the method, device and / or system described herein will be clear. For example, the order of operations described herein is only an example, and is not limited to those orders set forth herein, but can be changed as it will be clear after understanding the disclosure of the application, except for the operations that must occur in a specific order. In addition, for increased clarity and conciseness, the description of features known after understanding the disclosure of the application can be omitted.

[0043] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein that will be clear after understanding the disclosure of the present application.

[0044] Although terms such as "first", "second" and "third" may be used to describe various components, components, regions, layers or parts herein, these components, components, regions, layers or parts are not limited by these terms. Instead, these terms are only used to distinguish one component, component, region, layer or part from another component, component, region, layer or part. Therefore, without departing from the teachings of the examples described herein, the first component, first component, first region, first layer or first part referred to in the examples may also be referred to as the second component, second component, second region, second layer or second part.

[0045] The terms used herein are only used to describe various examples and will not be used to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the stated features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0046] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by ordinary technicians in the field to which the present disclosure belongs based on the understanding of the disclosure of the present application. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant field and the disclosure of the present application, and should not be interpreted in an idealized or overly formal way. The use of the term "may" with respect to examples or embodiments herein (for example, with respect to what an example or embodiment may include or implement) indicates that there is at least one example or embodiment that includes or implements such a feature, and all examples are not limited thereto.

[0047] Without special instructions, the same reference numerals generally refer to the same elements (e.g., components, steps, and methods). Reference numerals described in previous embodiments may be omitted if they appear again in later embodiments. In addition, the technical features described in different or the same embodiments may be combined in any manner, as long as the combined embodiments or technical solutions are complete and can solve the technical problems of the present application or achieve the technical effects described or not described in the present disclosure but can be determined based on the above complete technical solutions. The terms used in the present disclosure are explained below.

[0048] It should be noted that, in the absence of conflicts between the various embodiments, these embodiments and their features may be combined with each other.

[0049] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0050] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0051] Figure 1 is a flowchart illustrating a training method of a neural network based on time self-erasing according to an embodiment of the present disclosure.

[0052] Reference Figure 1 In step S110, a training sample including multimedia data is obtained. According to an embodiment of the present disclosure, the multimedia data may include at least one of image data, video data, etc., but is not limited thereto.

[0053] In step S120, for each time step in the multiple time steps, a first feature map corresponding to the time step is generated by inputting the acquired training samples into the convolution layer. According to an embodiment of the present disclosure, the neural network based on time self-erasing may be a spike neural network (SNN), but the training method of the neural network based on time self-erasing according to the present disclosure is not limited thereto. According to an embodiment of the present disclosure, the neural network based on time self-erasing may be used to implement classification, regression and / or target detection tasks, but is not limited thereto. According to an embodiment of the present disclosure, the neural network based on time self-erasing may include a convolution layer, the input of the convolution layer may be connected to the input of the neural network based on time self-erasing, and the first feature map may be the output of the convolution layer.

[0054] In step S130, a second feature map corresponding to the time step is generated by inputting the first feature map corresponding to the time step into the pooling layer. According to an embodiment of the present disclosure, the neural network based on time self-erasing may further include a pooling layer, the pooling layer may be a global average pooling layer, the input of the pooling layer may be connected to the output of the convolution layer, and the second feature map may be the output of the pooling layer. Figure 2 Step S130 is described in detail.

[0055] In step S140, a first prediction map corresponding to the time step is generated by inputting the second feature map corresponding to the time step into the first fully connected layer. According to an embodiment of the present disclosure, the neural network based on time self-erasure may further include a first fully connected layer, the input of the first fully connected layer may be connected to the output of the pooling layer, and the first prediction map may be the output of the first fully connected layer.

[0056] In step S150, based on the first prediction map corresponding to the time step and the true label, a loss corresponding to the time step is generated. Figure 3 Step S150 is described in detail.

[0057] In step S160, the parameters of the time self-erasing neural network are adjusted based on the loss corresponding to the time step to obtain a trained time self-erasing neural network.

[0058] Figure 2 is a flowchart illustrating a method of generating a second feature map according to an embodiment of the present disclosure.

[0059] Reference Figure 2, in step S131, for each time step except the first time step, an erasure mask is applied to a first feature map corresponding to the time step. In step S132, a second feature map corresponding to the time step is generated by inputting the first feature map to which the erasure mask is applied into a pooling layer. According to an embodiment of the present disclosure, the erasure mask may vary with each time step. Optionally, in step S133, for the first time step, a second feature map corresponding to the time step is generated by directly inputting the first feature map corresponding to the time step into a pooling layer.

[0060] In the previous time step, the predicted high probability score area corresponds to the semantically more discriminative area. Therefore, in the current time step, the erasure mask can be applied first before the first feature map is input to the pooling layer, so that the current time step pays more attention to other areas to mine more potential informative areas.

[0061] Figure 3 is a flow chart illustrating a method of generating losses according to an embodiment of the present disclosure.

[0062] Reference Figure 3 , in step S151, for the first time step, by applying the first prediction map corresponding to the time step and the true label to the cross entropy loss function, a loss corresponding to the time step is generated. In one example, step S151 can be shown as the following formula 1, where, represents the loss function, represents the cross entropy loss function, represents the prediction result of the first time step, represents the true label.

[0063] Formula 1:

[0064] In step S152, for each time step except the first time step, a sub-loss corresponding to the time step is generated by applying the first prediction map and the true label corresponding to the time step to the cross entropy loss function. In step S153, each sub-loss corresponding to all previous time steps before the time step is generated by applying the respective first prediction maps and the true labels corresponding to all previous time steps before the time step to the cross entropy loss function respectively. In step S154, the sum of the respective sub-losses corresponding to all previous time steps before the time step and the sub-loss corresponding to the time step is taken as the loss corresponding to the time step. In one example, steps S152 to S154 can be shown as the following formula 2, wherein, represents the time step, represents the current time step, represents the first prediction map generated in step S140 (ie, the first prediction map generated by sequentially inputting the first feature map to which the erasure mask is applied into the pooling layer and the first fully connected layer).

[0065] Formula 2:

[0066] Figure 4 is a flowchart illustrating a method of generating an erasure mask according to an embodiment of the present disclosure.

[0067] Reference Figure 4 In step S170, for each time step except the first time step, an average second prediction map corresponding to all previous time steps before the time step is generated. Figure 5 Step S170 is described in detail.

[0068] Figure 5 is a flowchart illustrating a method of generating an average second prediction map according to an embodiment of the present disclosure.

[0069] Reference Figure 5 In step S171, all first feature maps corresponding to all previous time steps before the time step are input into the second fully connected layer respectively, so as to generate respective second prediction maps corresponding to all previous time steps before the time step. According to an embodiment of the present disclosure, the difference between the second prediction map and the first prediction map is at least that: for each time step except the first time step, the second prediction map is not based on the erasure mask, while the first prediction map is based on the erasure mask.

[0070] In step S172, an average second prediction map corresponding to all previous time steps before the time step is generated by averaging the respective second prediction maps corresponding to all previous time steps before the time step.

[0071] Return to reference Figure 4 In step S180, a normalized average second prediction map corresponding to all previous time steps before the time step is generated by normalizing the average second prediction map corresponding to all previous time steps before the time step.

[0072] In one example, as shown in Formula 3 below, the probability scores of different categories corresponding to all previous time steps before the time step may be generated as a normalized average second prediction map corresponding to all previous time steps before the time step by applying a softmax function along the category dimension to the average second prediction map corresponding to all previous time steps before the time step, wherein, represents the normalized average second prediction map corresponding to all previous time steps before said time step, represents the index corresponding to the time step, Represents the second prediction graph.

[0073] Formula 3:

[0074] In step S190, an erasure mask is generated by suppressing regions associated with higher prediction probability scores based on a normalized average second prediction map corresponding to all previous time steps before the time step.

[0075] According to an embodiment of the present disclosure, for each position, when the value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is greater than a threshold, a first value is generated as the value of the erasure mask at the position. In one example, the first value may be 0. In addition, for each position, when the value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is less than the threshold, a second value different from the first value is generated as the value of the erasure mask at the position. In one example, the second value may be 1.

[0076] According to an embodiment of the present disclosure, the threshold includes a dynamic threshold and a fixed threshold, and the dynamic threshold varies based on the time step. In one example, the erasure mask may be calculated as shown in the following formula 4, where: represents the mask value of each position (i, j), represents the normalized average second prediction map value at each position (i, j), represents a fixed threshold, Represents the dynamic threshold, 0 means suppressing the area, and 1 means retaining the area.

[0077] Formula 4: According to an embodiment of the present disclosure, for each position, a dynamic threshold corresponding to the position can be generated based on the mean and standard deviation of the values ​​of all channels of the normalized average second prediction map corresponding to all previous time steps before the time step at the position.

[0078] In one example, as shown in the following formula 5, for each position, the product of the standard deviation and the hyperparameter and the sum of the mean value may be used as the dynamic threshold corresponding to the position, where: represents the normalized mean of the average second prediction map, represents the standard deviation of the normalized mean second prediction map, represents a hyperparameter.

[0079] Formula 5: According to the training method of the neural network based on time self-erasing according to the embodiment of the present disclosure, a time self-erasing mechanism is introduced to optimize the attention range of the neural network by gradually erasing redundant areas. This mechanism enables the network to focus on different feature areas at each time step, improves the diversity and learning efficiency of feature representation, and avoids the redundancy problem of feature representation in the time dimension in traditional neural networks. In addition, according to the training method of the neural network based on time self-erasing according to the embodiment of the present disclosure, an erasure mask is generated by combining a dynamic threshold with a fixed threshold, so that the neural network can adaptively suppress areas that have been considered important in the previous time step. This adaptive method improves the flexibility and effectiveness of the erasure mask and avoids the limitations of the fixed threshold strategy. Therefore, according to the training method of the neural network based on time self-erasing according to the embodiment of the present disclosure, the overall prediction accuracy is improved, the computational efficiency is improved, and the computational overhead is reduced.

[0080] Figure 6 is a flowchart illustrating a prediction method based on a time self-erasing neural network according to an embodiment of the present disclosure.

[0081] In step S210, data to be predicted including multimedia data is obtained. In step S220, a first prediction graph is generated as a prediction result by inputting the data to be predicted into a neural network based on time self-erasing. According to an embodiment of the present disclosure, the neural network based on time self-erasing is obtained by training according to the training method of the neural network based on time self-erasing described above. Therefore, the prediction method based on the neural network based on time self-erasing according to an embodiment of the present disclosure improves the overall prediction accuracy, improves the computational efficiency, and reduces the computational overhead.

[0082] Figure 7 is a block diagram illustrating a training device for a time-self-erasing neural network according to an embodiment of the present disclosure.

[0083] Reference Figure 7 , the training device 100 may include a sample acquisition module 110 , a training module 120 , a loss generation module 130 and a parameter adjustment module 140 .

[0084] The sample acquisition module 110 is configured to acquire training samples including multimedia data.

[0085] The training module 120 is configured to: for each time step among multiple time steps, generate a first feature map corresponding to the time step by inputting the acquired training samples into the convolution layer; generate a second feature map corresponding to the time step by inputting the first feature map corresponding to the time step into the pooling layer; generate a first prediction map corresponding to the time step by inputting the second feature map corresponding to the time step into the first fully connected layer.

[0086] According to an embodiment of the present disclosure, for each time step except the first time step, the training module 120 is configured to: apply an erasure mask to a first feature map corresponding to the time step. The erasure mask changes with each time step; and generate a second feature map corresponding to the time step by inputting the first feature map to which the erasure mask is applied into a pooling layer. For the first time step, the training module 120 is configured to: generate a second feature map corresponding to the time step by directly inputting the first feature map corresponding to the time step into the pooling layer.

[0087] The loss generation module 130 is configured to generate a loss corresponding to the time step based on the first prediction map corresponding to the time step and the true label.

[0088] According to an embodiment of the present disclosure, for the first time step, the loss generation module 130 is configured to: generate a loss corresponding to the time step by applying the first prediction map and the true label corresponding to the time step to the cross entropy loss function. In addition, for each time step other than the first time step, the loss generation module 130 is configured to: generate a sub-loss corresponding to the time step by applying the first prediction map and the true label corresponding to the time step to the cross entropy loss function; generate each sub-loss corresponding to all previous time steps before the time step by using each first prediction map and the true label corresponding to all previous time steps before the time step for the cross entropy loss function respectively; and use the sum of each sub-loss corresponding to all previous time steps before the time step and the sub-loss corresponding to the time step as the loss corresponding to the time step.

[0089] The parameter adjustment module 140 is configured to obtain a trained time-self-erasing neural network by adjusting the parameters of the time-self-erasing neural network based on the loss corresponding to the time step.

[0090] According to an embodiment of the present disclosure, the training device 100 may further include an erasure mask generation module 150. The erasure mask generation module 150 is configured to generate, for each time step except the first time step, an average second prediction map corresponding to all previous time steps before the time step.

[0091] According to an embodiment of the present disclosure, the erasure mask generation module 150 is configured to: generate respective second prediction maps corresponding to all previous time steps before the time step by inputting all first feature maps corresponding to all previous time steps before the time step into the second fully connected layer respectively; and generate an average second prediction map corresponding to all previous time steps before the time step by averaging the respective second prediction maps corresponding to all previous time steps before the time step.

[0092] According to an embodiment of the present disclosure, the erasure mask generation module 150 is configured to generate a normalized average second prediction map corresponding to all previous time steps before the time step by normalizing the average second prediction map corresponding to all previous time steps before the time step.

[0093] According to an embodiment of the present disclosure, the erasure mask generation module 150 is configured to: generate probability scores of different categories corresponding to all previous time steps before the time step as a normalized average second prediction map corresponding to all previous time steps before the time step by applying a flexible maximum function along the category dimension to the average second prediction map corresponding to all previous time steps before the time step.

[0094] According to an embodiment of the present disclosure, the erasure mask generation module 150 is configured to generate an erasure mask by suppressing regions associated with higher prediction probability scores based on a normalized average second prediction map corresponding to all previous time steps before the time step.

[0095] According to an embodiment of the present disclosure, the erasure mask generation module 150 is configured to: for each position, when the value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is greater than a threshold value, generate a first value as the value of the erasure mask at the position; for each position, when the value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is less than the threshold value, generate a second value different from the first value as the value of the erasure mask at the position.

[0096] According to an embodiment of the present disclosure, the threshold includes a dynamic threshold and a fixed threshold, and the dynamic threshold varies based on the time step. According to an embodiment of the present disclosure, the training device 100 may also include a dynamic threshold generation module 160. The dynamic threshold generation module 160 is configured to: for each position, based on the mean and standard deviation of the values ​​of all channels of the normalized average second prediction map corresponding to all previous time steps before the time step at the position, generate a dynamic threshold corresponding to the position.

[0097] According to an embodiment of the present disclosure, the dynamic threshold generation module 160 is configured to: for each position, take the product of the standard deviation and the hyperparameter and the sum of the mean as the dynamic threshold corresponding to the position.

[0098] According to the embodiment of the present disclosure, the training device based on the time self-erasing neural network introduces a time self-erasing mechanism to optimize the attention range of the neural network by gradually erasing redundant areas. This mechanism enables the network to focus on different feature areas at each time step, improves the diversity and learning efficiency of feature representation, and avoids the redundancy problem of feature representation in the time dimension in traditional neural networks. In addition, according to the embodiment of the present disclosure, the training device based on the time self-erasing neural network generates an erasure mask by combining a dynamic threshold with a fixed threshold, so that the neural network can adaptively suppress areas that have been considered important in the previous time step. This adaptive method improves the flexibility and effectiveness of the erasure mask and avoids the limitations of the fixed threshold strategy. Therefore, according to the embodiment of the present disclosure, the training device based on the time self-erasing neural network improves the overall prediction accuracy, improves the computational efficiency, and reduces the computational overhead.

[0099] Figure 8 is a block diagram illustrating a prediction device based on a time self-erasing neural network according to an embodiment of the present disclosure.

[0100] Reference Figure 8 , the prediction device 200 includes a data acquisition module 210 and a prediction module 220 .

[0101] The data acquisition module 210 is configured to: acquire the data to be predicted including multimedia data. The prediction module 220 is configured to: generate a first prediction graph as a prediction result by inputting the data to be predicted into a neural network based on time self-erasing. According to an embodiment of the present disclosure, the neural network based on time self-erasing is obtained by training according to the training device 100 based on time self-erasing neural network mentioned above. Therefore, the prediction device based on time self-erasing neural network according to an embodiment of the present disclosure improves the overall prediction accuracy, improves the computational efficiency, and reduces the computational overhead.

[0102] Fig. 9 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0103] Reference Fig. 9The electronic device 300 includes at least one processor 310 and at least one memory 320, wherein the at least one memory 320 stores computer executable instructions. When the computer executable instructions are executed by the at least one processor 310, the at least one processor 310 is prompted to execute the training method and / or prediction method of the time self-erasing neural network based on the embodiment of the present disclosure.

[0104] In the electronic device 300, the processor 310 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0105] The processor 310 may execute instructions or codes stored in the memory 320, which may also store data. The instructions and data may also be sent and received over the network via the network interface device, which may employ any known transmission protocol.

[0106] The memory 320 may be integrated with the processor 310, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 320 may include a separate device, such as an external disk drive, a storage array, or any other storage device that may be used by a database system. The memory 320 and the processor 310 may be operatively coupled or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 310 can read files stored in the memory.

[0107] Optionally, the electronic device 300 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 300 may be connected to each other via a bus and / or a network.

[0108] According to an embodiment of the present disclosure, a computer-readable storage medium may also be provided, wherein when the instructions stored in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the above-mentioned training method of a time-based self-erasing neural network or the above-mentioned prediction method of a time-based self-erasing neural network. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0109] According to an embodiment of the present disclosure, a computer program product may also be provided, comprising computer instructions, which, when executed by at least one processor, implement the above-mentioned training method of a time-self-erasing neural network or the above-mentioned prediction method of a time-self-erasing neural network.

[0110] According to the training method and training device of the neural network based on time self-erasing according to the embodiment of the present disclosure, a time self-erasing mechanism is introduced to optimize the attention range of the neural network by gradually erasing redundant areas. This mechanism enables the network to focus on different feature areas at each time step, improves the diversity and learning efficiency of feature representation, and avoids the redundancy problem of feature representation in the time dimension in traditional neural networks. In addition, according to the training device of the neural network based on time self-erasing according to the embodiment of the present disclosure, an erasure mask is generated by combining a dynamic threshold with a fixed threshold, so that the neural network can adaptively suppress the area that has been considered important in the previous time step. This adaptive method improves the flexibility and effectiveness of the erasure mask and avoids the limitations of the fixed threshold strategy. Therefore, according to the training method and device of the neural network based on time self-erasing according to the embodiment of the present disclosure, as well as the prediction method and device, the overall prediction accuracy is improved, the computational efficiency is improved, and the computational overhead is reduced.

[0111] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0112] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for a neural network based on time self-erasing, characterized in that: The training method comprises: obtaining training samples including multimedia data; For each time step of the plurality of time steps, generating a first feature map corresponding to the time step by inputting the acquired training sample into a convolutional layer; Generating a second feature map corresponding to the time step by inputting the first feature map corresponding to the time step into a pooling layer; Generating a first prediction map corresponding to the time step by inputting a second feature map corresponding to the time step into a first fully connected layer; Based on the first prediction map corresponding to the time step and the true label, generating a loss corresponding to the time step; By adjusting the parameters of the time self-erasing neural network based on the loss corresponding to the time step, a trained time self-erasing neural network is obtained, Wherein, for each time step except the first time step, the step of generating a second feature map corresponding to the time step by inputting the first feature map corresponding to the time step into the pooling layer includes: applying an erasure mask to a first feature map corresponding to the time step; By inputting the first feature map to which the erasure mask is applied into a pooling layer, a second feature map corresponding to the time step is generated, Here, the erasure mask changes with each time step.

2. The training method according to claim 1, characterized in that: The training method further comprises: generating, for each time step except the first time step, an average second prediction map corresponding to all previous time steps before the time step; generating a normalized average second prediction map corresponding to all previous time steps before the time step by normalizing the average second prediction map corresponding to all previous time steps before the time step; An erasure mask is generated by suppressing regions associated with higher prediction probability scores based on a normalized average second prediction map corresponding to all previous time steps before the time step.

3. The training method according to claim 2, characterized in that: The step of generating an average second prediction map corresponding to all previous time steps before the time step comprises: By inputting all first feature maps corresponding to all previous time steps before the time step into a second fully connected layer, respectively, generating respective second prediction maps corresponding to all previous time steps before the time step; An average second prediction map corresponding to all previous time steps before the time step is generated by averaging the respective second prediction maps corresponding to all previous time steps before the time step.

4. The training method according to claim 2, characterized in that: The step of generating a normalized average second prediction map corresponding to all previous time steps before the time step by normalizing the average second prediction map corresponding to all previous time steps before the time step comprises: By applying the flexible maximum function along the category dimension to the average second prediction map corresponding to all previous time steps before the time step, probability scores of different categories corresponding to all previous time steps before the time step are generated as a normalized average second prediction map corresponding to all previous time steps before the time step.

5. The training method according to claim 2, characterized in that: The step of generating an erasure mask by suppressing regions associated with higher prediction probability scores based on a normalized average second prediction map corresponding to all previous time steps before the time step comprises: For each position, when the value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is greater than a threshold, generate a first value as the value of the erasure mask at the position; For each position, when the value of the normalized average second prediction map corresponding to all previous time steps before the time step at the position is less than the threshold, a second value different from the first value is generated as the value of the erasure mask at the position.

6. The training method according to claim 1, characterized in that: For a first time step, the step of generating a second feature map corresponding to the time step by inputting the first feature map corresponding to the time step into a pooling layer includes: By directly inputting the first feature map corresponding to the time step into the pooling layer, a second feature map corresponding to the time step is generated.

7. The training method according to claim 1, characterized in that: For a first time step, based on a first prediction graph corresponding to the time step, the step of generating a loss corresponding to the time step comprises: generating a loss corresponding to the time step by applying a first prediction map corresponding to the time step and the true label to a cross entropy loss function, Wherein, for each time step except the first time step, based on the first prediction graph corresponding to the time step, the step of generating the loss corresponding to the time step comprises: Generate a sub-loss corresponding to the time step by applying a first prediction map corresponding to the time step and a true label to a cross entropy loss function; Generate respective sub-losses corresponding to all previous time steps before the time step by applying respective first prediction maps and true labels corresponding to all previous time steps before the time step to the cross entropy loss function, respectively; The loss corresponding to the time step is taken as the sum of the sub-losses corresponding to all previous time steps before the time step and the sub-loss corresponding to the time step.

8. A training device for a neural network based on time self-erasing, characterized in that: The training equipment comprises: The sample acquisition module is configured to: acquire a training sample including multimedia data; The training module is configured to: for each time step of the plurality of time steps, generate a first feature map corresponding to the time step by inputting the acquired training sample into the convolution layer; generate a second feature map corresponding to the time step by inputting the first feature map corresponding to the time step into the pooling layer; generate a first prediction map corresponding to the time step by inputting the second feature map corresponding to the time step into the first fully connected layer; A loss generation module is configured to: generate a loss corresponding to the time step based on a first prediction map corresponding to the time step and a true label; The parameter adjustment module is configured to: adjust the parameters of the time self-erasing neural network based on the loss corresponding to the time step to obtain a trained time self-erasing neural network, Here, for each time step except the first time step, the training module is configured as: applying an erasure mask to a first feature map corresponding to the time step; By inputting the first feature map to which the erasure mask is applied into a pooling layer, a second feature map corresponding to the time step is generated, Here, the erasure mask changes with each time step.

9. A computer-readable storage medium, characterized in that: When the instructions stored in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the training method of a time-self-erasing neural network according to any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by at least one processor, the training method of a neural network based on time self-erasing according to any one of claims 1 to 7 is implemented.

Images