Feature map processing method and data processing apparatus

Abstract

Embodiments of the present application disclose a feature map processing method and a data processing device in the field of artificial intelligence (AI), which can be applied to occasions where the feature map needs to be discarded in a neural network. The data processing device acquires a first feature map that needs to be discarded, determines autocorrelation information indicating the correlation degree between elements according to the first feature map, and determines the discarded output of the first feature map according to the autocorrelation information. The method determines how to discard the elements in the first feature map according to the correlation degree between the elements in the first feature map. Compared with the fixed probability setting in the traditional technology for discarding processing, the embodiments of the present application can discard part of the elements in the first feature map in a targeted manner, improve the generalization performance of the model, and make the discarded output retain more features conducive to model training, rather than completely randomly and irregularly discarding the elements in the first feature map.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a feature map processing method and a data processing apparatus. Background Technology

[0002] Overfitting often occurs during the training of neural network models, where the assumptions are made overly strict in an attempt to obtain a consistent hypothesis. When a trained network model exhibits overfitting, although it performs exceptionally well on the training set, it fails to generalize to new test sets. Therefore, regularization methods are needed in practical applications to process the network model and reduce the likelihood of overfitting. Neuron dropout is one such regularization technique for reducing overfitting.

[0003] In traditional neuron dropout techniques, the dropout probability is often set manually. During model training, some neurons in the neural network are randomly deactivated according to the aforementioned dropout probability, or the training output of some neurons is randomly set to zero according to the aforementioned dropout probability, thereby reducing the probability of model overfitting.

[0004] However, since the aforementioned discard probability is manually set, if the discard probability is not set properly, it can easily lead to insufficient or excessive discarding of input data during the discard operation, thus preventing the discard operation from achieving the expected results. Therefore, there is an urgent need for a discard processing method that can reasonably reduce the probability of model overfitting. Summary of the Invention

[0005] This application provides a feature map processing method and a data processing device for discarding attributes of feature maps to more reasonably reduce the probability of model overfitting and improve model generalization performance.

[0006] Firstly, a feature map processing method is provided, which can be applied to situations in neural networks where feature maps need to be discarded. In this method, a data processing device first acquires a first feature map to be discarded. This first feature map can be an initial matrix (e.g., a matrix determined by first image data, or a matrix determined by first audio data), or a feature map processed by a convolutional layer or pooling layer. The first feature map includes multiple elements and reflects the distribution characteristics of these elements. Then, the data processing device determines autocorrelation information based on the first feature map. This autocorrelation information indicates the degree of correlation between the elements. Finally, the data processing device determines the discard output of the first feature map based on the autocorrelation information. In other words, the data processing device determines how to discard elements in the first feature map based on the degree of correlation between the elements.

[0007] Here, "discarding output" refers to the remaining part of the first feature map after discarding some features (i.e., one or more elements in the first feature map). Furthermore, in this embodiment, "discarding" means changing the value of an element in the feature map from a non-zero value to 0, or changing the value of an element in the feature map to a non-zero value.

[0008] In this embodiment, the data processing device performs a discard operation on the first feature map based on the correlation determined by the first feature map. That is, the discard operation in this embodiment considers the correlation degree of each element in the first feature map rather than discarding elements randomly. Since discarding some elements in the first feature map can reduce the probability of model overfitting, performing the discard operation on the first feature map according to its autocorrelation information makes the discard operation more reasonable while reducing model overfitting, and makes the discarded output more likely to achieve the expected effect. Compared to the traditional technique of setting a fixed probability to discard neurons, the solution provided in this embodiment discards elements based on the attributes of the feature map corresponding to the data itself. This allows for targeted discarding of some elements in the first feature map (which can be understood as the output of a neuron after processing the input data). For first feature maps from different data sources, it can adaptively discard elements based on the characteristics of the first feature map corresponding to each data source. While improving the generalization performance of the neural network model, it allows the discarded output to retain more features beneficial to model training, rather than randomly discarding elements in the first feature map.

[0009] In one possible implementation, determining the discard output of the first feature map based on the autocorrelation information includes: when the autocorrelation information indicates that the correlation between the elements is greater than a threshold, performing a unidirectional operation on the elements, the unidirectional operation including discarding the elements or retaining the elements; and when the autocorrelation information indicates that the correlation between the elements is less than the threshold, performing a reversible operation on the elements, the reversible operation including discarding one element and retaining the other element.

[0010] In this embodiment, autocorrelation information is used to indicate the degree of correlation between elements in the first feature graph. When the degree of correlation indicated by the autocorrelation information of two or more elements in the first feature graph is greater, the dependency between the two or more elements is stronger. Therefore, discarding or missing one element has a greater impact on the other element, and the same operation should be performed on the two or more elements during the calculation. For example, two or more elements with high correlation are retained simultaneously, or two or more elements with high correlation are discarded simultaneously. Conversely, when the degree of correlation indicated by the autocorrelation information of two or more elements in the first feature graph is smaller, the dependency between the two or more elements is weaker, and discarding or missing one element has a smaller impact on the other element. In this case, different operations are performed on the two elements in the first feature graph. For example, when the correlation between two elements is low, if one element is retained, the other element is discarded.

[0011] This embodiment proposes a different discarding strategy based on the varying degrees of relevance indicated by autocorrelation information. In other words, the discard probability differs depending on the autocorrelation information of the feature map. The discard probability is determined by the autocorrelation information of the feature map; when the autocorrelation information of the feature map changes, the discard probability also changes, and the discarding operation on the first feature map will adaptively change accordingly. Therefore, the discarding scheme in this embodiment can achieve adaptive discarding based on the autocorrelation information of the first feature map, improving the generalization performance of the neural network model.

[0012] In one possible implementation, determining the discard output of the first feature map based on the autocorrelation information includes: determining a discard mask based on the autocorrelation information, the discard mask indicating the element to be discarded; and then determining a second feature map based on the discard mask and the first feature map, the second feature map being the discard output.

[0013] In this embodiment, the discard mask used for discarding elements in the first feature map is determined based on the autocorrelation information of the first feature map. Since this autocorrelation information indicates the degree of correlation between elements in the first feature map, the discard mask can adaptively change as the elements of the first feature map change. In other words, the discard mask can be adaptively adjusted according to the features of the first feature map, rather than being a fixed value set manually. Therefore, this scheme not only reduces the probability of model overfitting during the discarding process but also selectively discards elements in the first feature map, ensuring that the discarded output retains more features beneficial to model training, rather than randomly and irregularly discarding elements from the first feature map.

[0014] In one possible implementation, the autocorrelation information includes semantic-based autocorrelation information and / or spatial-based autocorrelation information. The semantic-based autocorrelation information is the correlation information determined based on the numerical value of each element in the first feature map. The spatial-based autocorrelation information is the correlation information determined based on the spatial position of each element in the first feature map.

[0015] In one possible implementation, the autocorrelation information is a correlation coefficient matrix. Determining the drop mask based on the autocorrelation information includes: normalizing the correlation coefficient matrix to obtain a first probability matrix; obtaining a first random number matrix, the size of which is the same as the size of the first probability matrix; using the first probability matrix as a threshold for binarization, and binarizing the random matrix to obtain the drop mask.

[0016] Optionally, the sum of the probability values ​​in each row or column of the first probability matrix is ​​1.

[0017] In this embodiment, a first probability matrix is ​​used to binarize the random number matrix to obtain a discard mask. When this discard mask is applied to the feature map, it introduces a perturbation to the elements in the feature map. However, this perturbation is not the random perturbation used in traditional techniques, but rather a perturbation generated according to the features in the feature map. Therefore, it is possible to discard a portion of the elements in the feature map while preserving as many features as possible that are more beneficial to model training. In other words, the entire discarding process is based on the attributes or features of the feature map, rather than being completely random.

[0018] In one possible implementation, the discard mask is used to zero out or modify some elements in the first feature map.

[0019] In one possible implementation, determining the second feature map based on the drop mask and the first feature map includes: obtaining a third feature map based on the product of the drop mask and the first feature map; and obtaining the second feature map by subtracting the first feature map from the third feature map.

[0020] In this embodiment, it is proposed to directly use the discard mask to perform a discard operation on the first feature map in order to discard or modify some elements in the first feature map.

[0021] In one possible implementation, determining the second feature map based on the drop mask and the first feature map includes: obtaining a fifth feature map based on the product of the drop mask and the fourth feature map, wherein the fourth feature map is the product of the first probability matrix and the first feature map, or the fourth feature map is the product of the first probability matrix and the first feature map with a linear or nonlinear transformation; and obtaining the second feature map by subtracting the first feature map from the fifth feature map.

[0022] In this embodiment, the fourth feature map is obtained by transforming the first feature map according to the first probability matrix, or by transforming the transformation matrix (e.g., a linear or nonlinear transformation matrix) of the first feature map according to the first probability matrix. Therefore, the fourth feature map carries most of the features of the first feature map, but it is not exactly the same as the first feature map. Therefore, multiplying the discard mask by the fourth feature map is equivalent to perturbing the first feature map. The method of determining the second feature map (i.e., discarding the output) by subtracting the first feature map from the fifth feature map can introduce perturbation to the first feature map, and can retain as many features in the first feature map as possible that are more conducive to model training while discarding some elements.

[0023] In one possible implementation, the correlation coefficient matrix includes a spatial correlation coefficient matrix, which indicates the degree of correlation between the positions of different elements in the first feature map in the spatial domain.

[0024] In this embodiment, a spatial correlation coefficient matrix is ​​proposed to quantify the autocorrelation information. This allows for the determination of the autocorrelation information of the first feature map not only by measuring its degree but also by measuring the aforementioned autocorrelation information with accurate numerical values.

[0025] In one possible implementation, determining the autocorrelation information of the first feature map includes multiplying the transpose of the first feature map with the first feature map to obtain the spatial correlation coefficient matrix.

[0026] This embodiment proposes a method for calculating the spatial correlation coefficient. However, when using the formula for calculation, a linear transformation may be performed on the aforementioned first feature map for computational convenience. For example, a 1×1 convolution operator may be used to process the aforementioned first feature map, and then the transpose of the processed first feature map may be multiplied with the first feature map to obtain the spatial correlation coefficient matrix.

[0027] In one possible implementation, the first feature map is an initial matrix; or, the first feature map is obtained by convolution operation on the initial matrix at least once; wherein, one element of the first feature map corresponds to at least one pixel in the initial matrix, and the initial matrix includes a portion of the first image data or the first audio data.

[0028] In one possible implementation, the first feature map is the initial matrix, which includes a portion of the first image data or the first audio data.

[0029] In one possible implementation, the dropout mask is located in a dropout processing layer in a neural network, the first feature map is the input of the dropout processing layer, the second feature map is the output of the dropout processing layer, the first feature map is the output of the layer above the dropout processing layer, and the second feature map is the input of the layer below the dropout processing layer.

[0030] In one possible implementation, the upper layer is the input layer of the neural network model; and / or, the lower layer is the output layer of the neural network model.

[0031] In one possible implementation, the upper layer and / or the lower layer are hidden layers in the neural network model.

[0032] It should be noted that there are many other specific implementation methods in this application, and the specific implementation methods and their beneficial effects in the first aspect can be found therein, which will not be repeated here.

[0033] Secondly, a data processing apparatus is provided, the data processing apparatus comprising:

[0034] The acquisition module is used to acquire a first feature map corresponding to the first image data or the first audio data. The first feature map includes multiple elements and is used to indicate the distribution characteristics of the multiple elements. The calculation module is used to determine the autocorrelation information of the first feature map and is used to indicate the degree of correlation between each element. The calculation module is also used to determine the discard output corresponding to the first feature map based on the autocorrelation information. The discard output is the data of the first feature map after discarding.

[0035] Here, "discarding output" refers to the remaining part of the first feature map after discarding some features (i.e., one or more elements in the first feature map). Furthermore, in this embodiment, "discarding" means changing the value of an element in the feature map from a non-zero value to 0, or changing the value of an element in the feature map to a non-zero value.

[0036] In this embodiment, the data processing device performs a discard operation on the first feature map based on the correlation determined by the first feature map. That is, the discard operation in this embodiment considers the correlation degree of each element in the first feature map rather than discarding elements randomly. Since discarding certain elements in the first feature map can reduce the probability of model overfitting, performing the discard operation on the first feature map according to its autocorrelation information makes the discard operation more reasonable while reducing model overfitting, thus increasing the probability that the discarded output will achieve the expected results. Compared to traditional techniques that set a fixed probability for discarding, this allows for targeted discarding of certain elements in the first feature map. While improving the model's generalization performance, it also allows the discarded output to retain more features beneficial to model training, rather than randomly discarding elements in the first feature map.

[0037] In one possible implementation, the calculation module is specifically configured to: when the autocorrelation information indicates that the correlation between the elements is greater than a threshold, perform a unidirectional operation on the elements, the unidirectional operation including discarding the elements or retaining the elements; and when the autocorrelation information indicates that the correlation between the elements is less than the threshold, perform a reverse operation on the elements, the reverse operation including discarding one element and retaining the other element.

[0038] In this embodiment, autocorrelation information is used to indicate the degree of correlation between elements in the first feature graph. When the degree of correlation indicated by the autocorrelation information of two or more elements in the first feature graph is greater, the dependency between the two or more elements is stronger. Therefore, discarding or missing one element has a greater impact on the other element, and the same operation should be performed on the two or more elements during the calculation. For example, two or more elements with high correlation are retained simultaneously, or two or more elements with high correlation are discarded simultaneously. Conversely, when the degree of correlation indicated by the autocorrelation information of two or more elements in the first feature graph is smaller, the dependency between the two or more elements is weaker, and discarding or missing one element has a smaller impact on the other element. In this case, different operations are performed on the two elements in the first feature graph. For example, when the correlation between two elements is low, if one element is retained, the other element is discarded.

[0039] This embodiment proposes determining different discarding strategies based on the varying degrees of relevance indicated by autocorrelation information. In other words, different autocorrelation information in the feature map results in different discard probabilities. These discard probabilities are determined by the autocorrelation information of the feature map; when the autocorrelation information of the feature map changes, the discard probability also changes, and the discarding operation on the first feature map will adaptively change accordingly. Therefore, the discarding scheme in this embodiment can achieve adaptive discarding based on the autocorrelation information of the first feature map.

[0040] In one possible implementation, determining the discard output of the first feature map based on the autocorrelation information includes: determining a discard mask based on the autocorrelation information, the discard mask indicating the element to be discarded; and then determining a second feature map based on the discard mask and the first feature map, the second feature map being the discard output.

[0041] In this embodiment, the discard mask used for discarding elements in the first feature map is determined based on the autocorrelation information of the first feature map. Since this autocorrelation information indicates the degree of correlation between elements in the first feature map, the discard mask can adaptively change as the elements of the first feature map change. In other words, the discard mask can be adaptively adjusted according to the features of the first feature map, rather than being a fixed value set manually. Therefore, this scheme not only reduces the probability of model overfitting during the discarding process but also selectively discards elements in the first feature map, ensuring that the discarded output retains more features beneficial to model training, rather than randomly and irregularly discarding elements from the first feature map.

[0042] In one possible implementation, the autocorrelation information includes semantic-based autocorrelation information and / or spatial-based autocorrelation information. The semantic-based autocorrelation information is the correlation information determined based on the numerical value of each element in the first feature map. The spatial-based autocorrelation information is the correlation information determined based on the spatial position of each element in the first feature map.

[0043] In one possible implementation, the autocorrelation information is a correlation coefficient matrix. The calculation module is specifically used to: normalize the correlation coefficient matrix to obtain a first probability matrix; obtain a first random number matrix, the size of which is the same as the size of the first probability matrix; and use the first probability matrix as a threshold for binarization to binarize the random matrix, thereby obtaining the discard mask.

[0044] Optionally, the sum of the probability values ​​in each row or column of the first probability matrix is ​​1.

[0045] In this embodiment, a first probability matrix is ​​used to binarize the random number matrix to obtain a discard mask. When this discard mask is applied to the feature map, it introduces a perturbation to the elements in the feature map. However, this perturbation is not the random perturbation used in traditional techniques, but rather a perturbation generated according to the features in the feature map. Therefore, it is possible to discard a portion of the elements in the feature map while preserving as many features as possible that are more beneficial to model training. In other words, the entire discarding process is based on the attributes or features of the feature map, rather than being completely random.

[0046] In one possible implementation, the discard mask is used to zero out or modify some elements in the first feature map.

[0047] In one possible implementation, the calculation module is specifically used to: obtain a third feature map based on the product of the discard mask and the first feature map; and obtain the second feature map by subtracting the first feature map from the third feature map.

[0048] In one possible implementation, the calculation module is specifically used to: obtain a fifth feature map based on the product of the discard mask and the fourth feature map, wherein the fourth feature map is the product of the first probability matrix and the first feature map, or the fourth feature map is the product of the first probability matrix and the linear or nonlinear transformation of the first feature map; and obtain the second feature map by subtracting the first feature map from the fifth feature map.

[0049] In one possible implementation, the correlation coefficient matrix includes a spatial correlation coefficient matrix, which indicates the degree of correlation between the positions of different elements in the first feature map in the spatial domain.

[0050] In one possible implementation, the calculation module is specifically used to: multiply the transpose of the first feature map by the first feature map to obtain the spatial correlation coefficient matrix.

[0051] In one possible implementation, the first feature map is an initial matrix; or the first feature map is obtained by at least one convolution operation; wherein one element of the first feature map corresponds to at least one pixel in the initial matrix, and the initial matrix includes a portion of the first image data or the first audio data.

[0052] In one possible implementation, the first feature map is the initial matrix, which includes a portion of the first image data or the first audio data.

[0053] In one possible implementation, the dropout mask is located in a dropout processing layer in a neural network, the first feature map is the input of the dropout processing layer, the second feature map is the output of the dropout processing layer, the first feature map is the output of the layer above the dropout processing layer, and the second feature map is the input of the layer below the dropout processing layer.

[0054] In one possible implementation, the upper layer is the input layer of the neural network model; and / or, the lower layer is the output layer of the neural network model.

[0055] In one possible implementation, the upper layer and / or the lower layer are hidden layers in the neural network model.

[0056] It should be noted that there are many other specific implementation methods in this application, and the specific implementation methods and their beneficial effects in the first aspect can be found therein, which will not be repeated here.

[0057] Thirdly, a data processing apparatus is provided, which may be a general-purpose computer, a large server, or an analytical device, or other devices with analytical or computational functions. The data processing apparatus includes at least one processor and at least one memory. The processor may be a general-purpose central processing unit or a microprocessor; the processor may be a single-core processor or a multi-core processor. Furthermore, the processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). The processor is capable of calling program code in memory to implement the functions described in the first aspect and any embodiment of the first aspect.

[0058] Optionally, the data processing apparatus further includes a communication interface for communicating with other servers or network devices. The data processing apparatus can receive instructions or data (e.g., a first feature map or an initial matrix) from other devices through this communication interface. For example, the communication interface can be a transceiver, such as a wireless transceiver or an optical transceiver.

[0059] Fourthly, a model training system is provided, which includes the data processing device described in the second aspect above. The feature processing device can be used as part of the model training system so that the model training system can train a network model with good generalization ability.

[0060] Fifthly, a computer-readable storage medium is provided that stores a computer program or instructions thereon, which, when executed, cause a data processing apparatus to perform the methods described in the first aspect and any embodiment of the first aspect.

[0061] In a sixth aspect, a computer program product comprising instructions is provided, the computer program product including computer program code, which, when run on a computer, causes the computer to perform the methods described in the first aspect and any embodiment of the first aspect.

[0062] In a seventh aspect, a digital processing chip is provided. This digital processing chip integrates circuitry for implementing the aforementioned processor or processor functions, and one or more interfaces. When the digital processing chip integrates a memory, it can perform the method steps described in any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface to obtain data to be processed from the external memory. Furthermore, the digital processing chip can also implement the actions performed by the data processing device in the foregoing embodiments based on program code stored in the external memory.

[0063] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0064] In this embodiment, the data processing device performs a discard operation on the first feature map based on the correlation determined by the first feature map. That is, the discard operation in this embodiment considers the correlation degree of each element in the first feature map rather than discarding elements randomly. Since discarding certain elements in the first feature map can reduce the probability of model overfitting, performing a discard operation on the first feature map according to its autocorrelation information makes the discard operation more reasonable while reducing model overfitting, thus increasing the probability that the discarded output will achieve the expected effect. Compared to traditional techniques that set a fixed probability for discarding, this allows for targeted discarding of certain elements in the first feature map. While improving the model's generalization performance, it also allows the discarded output to retain more features beneficial to model training, rather than randomly discarding elements in the first feature map. Attached Figure Description

[0065] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application.

[0066] Figure 1 This is a system architecture diagram applicable to the feature map processing method in the embodiments of this application;

[0067] Figure 2A This is a schematic diagram of a convolutional neural network structure;

[0068] Figure 2B This is a schematic diagram of another convolutional neural network structure;

[0069] Figure 2C This is a schematic diagram of a neural network structure in an embodiment of this application;

[0070] Figure 2D This is a schematic diagram of another neural network structure in an embodiment of this application;

[0071] Figure 2E This is a schematic diagram of another neural network structure in an embodiment of this application;

[0072] Figure 3 This is a flowchart of a feature map processing method in an embodiment of this application;

[0073] Figure 4A This is an example diagram of how the first feature map is determined by the initial matrix in an embodiment of this application;

[0074] Figure 4B This is another example diagram illustrating the determination of the first feature map by the initial matrix in this embodiment of the application;

[0075] Figure 4C This is an example diagram illustrating the determination of the first feature map from the feature map in an embodiment of this application;

[0076] Figure 4D This is an example diagram illustrating the relationship between elements in the first feature map and pixels in the initial matrix in an embodiment of this application.

[0077] Figure 4E This is an example diagram illustrating the determination of the correlation coefficient matrix in an embodiment of this application;

[0078] Figure 5A This is another example diagram illustrating the determination of the correlation coefficient matrix in the embodiments of this application;

[0079] Figure 5B This is an example diagram of the normalization process in the embodiments of this application;

[0080] Figure 5C This is an example diagram illustrating the determination of the discard mask in an embodiment of this application;

[0081] Figure 6A This is a schematic diagram of an embodiment of determining a second feature map based on a first feature map and a discard mask in this application.

[0082] Figure 6B This is an example diagram of determining the third feature map in an embodiment of this application;

[0083] Figure 6C This is an example diagram illustrating the determination of the second feature map in an embodiment of this application;

[0084] Figure 7AThis is a schematic diagram of another embodiment of determining the second feature map based on the first feature map and the discard mask in this application;

[0085] Figure 7B This is an example diagram of determining the fourth feature map in an embodiment of this application;

[0086] Figure 7C This is an example diagram for determining the fifth feature map in an embodiment of this application;

[0087] Figure 7D This is another example diagram for determining the second feature map in the embodiments of this application;

[0088] Figure 8 This is a schematic diagram of one embodiment of the data processing device in this application;

[0089] Figure 9 This is a schematic diagram of an embodiment in which the data processing device is implemented by a chip in this application.

[0090] Figure 10 This is a schematic diagram of another embodiment of the data processing device in this application. Detailed Implementation

[0091] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0092] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0093] This application provides a feature map processing method and a data processing device for discarding attributes of feature maps, which can reasonably reduce the probability of model overfitting and improve model generalization performance.

[0094] To facilitate understanding, some of the technical terms involved in the embodiments of this application will be explained below:

[0095] Overfitting refers to a situation where, during the model parameter fitting process, the data used for fitting contains sampling errors. This causes the model training process to account for these sampling errors, resulting in an overly rigorous model. Consequently, when training the model using the training set (i.e., the data used for fitting), the output is good; however, when training the model using other datasets, the output is poor. This situation is called model overfitting.

[0096] Training set: refers to the data samples used for model fitting. In this embodiment, a sample in the training set is called a training sample. This training sample is matrix data, and its actual form varies in different scenarios. For example, in image or video processing scenarios, the training sample is an image; in data point processing scenarios, the training sample can be a matrix composed of data points arranged according to a certain rule, i.e., matrix data. Specifically, this embodiment does not limit the actual form of the training sample.

[0097] Validation set (development set): This refers to a dataset set aside separately during model training. This validation set is typically used to tune the model's hyperparameters and to conduct an initial evaluation of the model's capabilities.

[0098] Test set: refers to the dataset used to evaluate the generalization ability of the final model.

[0099] Generalization ability refers to a model's ability to adapt to new sample data. Generally, a model is first fitted using a training set, then evaluated using a validation set to adjust the model's hyperparameters, and finally tested using a test set to verify the model's generalization ability. Since the test set typically consists of new sample data—data that has never been used in model training—if training the model using this test set yields suitable training results, then the model is said to have good generalization ability.

[0100] Feature map: Also known as feature layer, representation feature, feature map, or network representation. This feature map can be another matrix obtained by performing a convolution operation on an initial matrix using a feature extractor or feature extraction operator (e.g., a convolution kernel). Each element in this feature map is a numerical value. Features extracted by performing convolution operations on the same initial matrix using different convolution kernels are also possible. Figure 1 The difference is not always apparent. When the aforementioned initial matrix is ​​an image, it can be understood that the feature map is a matrix that reflects the features of an image obtained through convolution operations.

[0101] Receptive field: refers to the size (or range) of the region on the training sample that an element on the feature map of the output of each layer in a convolutional neural network maps to. In other words, one element on the feature map corresponds to a specific region on the training sample.

[0102] Normalization is a preprocessing method proposed for the convenience of data processing. Generally, normalization transforms a set of data to the same dimension and maps each data point in the set to the interval [0,1]. It can also be understood as compressing a set of data into the range [0,1]. After normalization, the speed of gradient descent in finding the optimal solution can be accelerated. The normalization process mentioned in this embodiment compresses each row (or column) element of the matrix, so that after processing, each element of that row (or column) lies in the interval [0,1]. Optionally, the cumulative sum of all elements in that row (or column) is 1.

[0103] Discarded mask: A matrix consisting of 0s and 1s. When this discarded mask is multiplied by the feature map, some elements in the feature map are set to zero or modified. The product of the discarded mask and the feature map represents the elements that should be discarded from the original feature map and the degree of discarding.

[0104] Discarded output: Also known as filtered output, this refers to the data retained after the input data has undergone a discard operation. In this embodiment, it refers to the remaining portion of the first feature map after discarding some features (i.e., one or more elements in the first feature map). Furthermore, in this embodiment, discarding means changing the value of an element in the feature map from a non-zero value to 0, or changing the value of an element in the feature map to a non-zero value.

[0105] The following describes the system architecture applicable to the feature map processing method and data processing device (e.g., in one implementation, it can be called a feature map processing device) proposed in the embodiments of this application:

[0106] like Figure 1The diagram illustrates a system architecture applicable to a feature map processing method and a data processing device. This system architecture includes a data processing device 001, a database 002, a client device 003, and a data acquisition device 004. The data processing device 001 acquires training samples from either the client device 003 or the database 002, and then constructs a training model based on these training samples. Furthermore, the training samples in the database 002 can originate from either the client device 003 or the data acquisition device 004. The data acquisition device 004 collects data and stores it in the database 002. Additionally, the database 002 may also store feature maps output by historical training models. In this case, the data processing device 001 can acquire feature maps from the database 002 and then construct a training model based on the previous feature maps.

[0107] The data processing device 001 includes a computing module 011, a transceiver module 012, and a storage module 013. The data processing device 001 can receive training samples from the client device 003 via the transceiver module 012. The computing module 012 can also invoke computer program code from the storage module 013 to execute a feature map processing method to obtain a training model capable of outputting a second feature map based on a first feature map. See below for details. Figure 3 The relevant descriptions in the corresponding embodiments.

[0108] Furthermore, the aforementioned transceiver module 012 can be a wireless transceiver, an optical transceiver, or a wired interface (such as an I / O interface), as well as other devices capable of data interaction with external devices. The data processing device 001 can also return the constructed training model to the client device 003 through the transceiver module 012, so that the client device 003 may deploy the aforementioned training model in the client device 003 or other devices.

[0109] In this embodiment, the aforementioned data processing device 001 can be implemented by one or more servers with computing capabilities; or, it can be implemented by one or more management servers in conjunction with other computing devices. Furthermore, the aforementioned system architecture may also include devices such as data storage, routers, and load balancers. The data processing device 001 can be deployed at one physical site within this system architecture, or distributed across multiple physical sites.

[0110] Furthermore, users can interact with the data processing device 001 using their respective client devices 003. The client device 003 can be a personal computer, computer workstation, smartphone, tablet computer, or smart camera, etc. The aforementioned client device 003 can interact with the data processing device 001 via a communication network using any communication mechanism / standard. This communication network can be a wide area network (WAN), a local area network (LAN), or a point-to-point connection, etc. For example, the communication network can include a wireless network, a wired network, or a combination of wireless and wired networks, etc. For example, the wireless network includes, but is not limited to, any one or more combinations of: 5th-generation (5G) systems, Long Term Evolution (LTE) systems, Global System for Mobile Communication (GSM), Code Division Multiple Access (CDMA) networks, Wideband Code Division Multiple Access (WCDMA) networks, Wireless Fidelity (Wi-Fi), Bluetooth, Zigbee, Radio Frequency Identification (RFID), long-range (Lora) wireless communication, and near-field communication (NFC). The wired network may include fiber optic communication networks or networks composed of coaxial cables, etc. Specific limitations are not specified here.

[0111] It should be noted that, Figure 1 This is merely an example of a system architecture provided in this application. In practical applications, the system to which the feature map processing method is applicable includes, in addition to, [other systems]. Figure 1 In addition to the devices, components, and modules shown, other modules or components may also be included, which are not limited here. In this embodiment and subsequent embodiments, the aforementioned system architecture is used as an example for description only.

[0112] The following describes the application scenarios applicable to the feature map processing method and data processing device proposed in the embodiments of this application:

[0113] The feature map processing method proposed in this application is mainly applied to neural networks containing one or more hidden layers. During the training process of a neural network-based model, the feature map output by the previous layer of the neural network can be processed, causing some features of the output feature map to be discarded. The resulting discarded feature map is then input into the next layer of the neural network for further model training.

[0114] It should be understood that the training model mentioned in the embodiments of this application can be a prediction model, a classification model, an anomaly detection model, or other models that involve discarding elements or features in the feature map; no specific limitation is made here. In this embodiment and subsequent embodiments, only the training model is used as an example for description.

[0115] It should also be understood that the neural networks mentioned in the embodiments of this application can include various types. For example, convolutional neural networks (CNN), deep neural networks (DNN), recurrent neural networks (RNN), or other neural networks. No specific limitation is made here. In this embodiment and subsequent embodiments, only convolutional neural networks are used as an example for description.

[0116] To facilitate understanding of the subsequent feature map processing methods, we will first introduce the network structure of Convolutional Neural Networks (CNNs).

[0117] A Convolutional Neural Network (CNN) is a deep neural network with convolutional structures, and is a deep learning architecture. This deep learning architecture refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network where each neuron responds to overlapping regions in the input training samples (e.g., images or matrices of discrete data points). A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input training sample, outputting one or more feature maps. Generally, the number of feature maps is determined by the number of convolutional kernels, and the number of feature maps equals the number of convolutional kernels. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input training samples. In a convolutional layer of a CNN, a neuron may only be connected to a subset of its neighboring neurons. In the same convolutional layer, multiple convolutional kernels can be used to extract different features or information from the training samples. Generally, the more convolutional kernels there are, the richer the features or information of the training samples reflected by the convolutional operation.

[0118] like Figure 2A As shown, the convolutional neural network (CNN) 100 may include an input layer 110, a convolutional / pooling layer 120, and a neural network layer 130. The neural network layer 130 includes at least one hidden layer and one output layer.

[0119] The convolutional / pooling layers 120 may include layers 121-126 as shown in Examples 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer; in another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation. It should be noted that the aforementioned pooling layers are optional. That is, layers 121-126 can be all convolutional layers, or pooling layers can be inserted between two convolutional layers.

[0120] Taking convolutional layer 121 as an example, convolutional layer 121 can include one or more convolution operators, also called kernels. Using images as training samples, the convolution kernel acts as a filter to extract specific information from the input image matrix (i.e., a matrix composed of pixel values) during image processing. The convolution operator can essentially be a weight matrix, which is usually predefined. During the convolution operation, the convolution kernel typically processes the input image along the horizontal direction, N pixels at a time, to extract specific features. Here, N is an integer greater than or equal to 1. However, the size of the convolution kernel should be related to the image size. Specifically, the size of the convolution kernel is less than or equal to the image size. It is important to note that the depth dimension of the convolution kernel is the same as the depth dimension of the input image; during the convolution operation, the convolution kernel extends to the entire depth of the input image. Therefore, convolving the input image with a single convolution kernel produces a single-depth convolutional output. However, in most cases, a single convolutional kernel is not used; instead, multiple convolutional kernels of the same dimension are applied. The outputs of each kernel are stacked to form the depth dimension of the convolutional image. Different kernels can be used to extract different features from the image. For example, one kernel can be used to extract edge information, another to extract specific colors, and yet another to blur unwanted noise. Since these multiple kernels have the same dimension, the feature maps extracted by them also have the same dimension. These extracted feature maps are then merged to form the output of the convolution operation.

[0121] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0122] Furthermore, because it is often necessary to reduce the number of training parameters, pooling layers are frequently introduced periodically after convolutional layers. That is, for example... Figure 2ALayers 121-126 in example 120 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range. The max pooling operator takes the pixel value with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the convolution kernel in a convolutional layer should be related to the image size, the operators in a pooling layer are also related to the image size. The size of the output image after processing by the pooling layer can be smaller than the size of the input image of the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0123] Furthermore, after processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required information. This is because the convolutional / pooling layers 120 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize neural network layers 130 to generate one or a set of required class numbers of output. Therefore, neural network layers 130 may include multiple hidden layers (such as...). Figure 2A As shown in 131, 132 to 13n) and output layer 140.

[0124] In this application, if the feature map processing method provided in the various embodiments of this application is abstracted into a new layer (such as a hidden layer), then the new hidden layer can be located in any one of the multiple hidden layers in the neural network layer 130. Alternatively, it can be understood that the new hidden layer can be inserted between any two hidden layers in the neural network layer 130. Furthermore, a layer abstracted from this feature map processing method can also be located after a convolutional layer. If the feature map processing method provided in the various embodiments of this application is abstracted into an operator, then this operator can be added to any layer in a neural network or replace any operator originally used for discarding feature maps in any layer. Specific limitations are not specified here.

[0125] After the multiple hidden layers in neural network layer 130, the final layer of the entire convolutional neural network 100 is the output layer 140. This output layer 140 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 100 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 2A The propagation from 110 to 140 is completed (forward propagation), and the reverse propagation (such as...) Figure 2AThe propagation from 140 to 110 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the output of the convolutional neural network 100 through the output layer and the ideal result.

[0126] It should be noted that, as Figure 2A The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models. For example, such as... Figure 2B The convolutional / pooling layer 120 shown can also contain multiple convolutional or pooling layers in parallel. The features extracted by the parallel convolutional or pooling layers are input into the neural network layer 130 for processing, so that the output layer 140 outputs the final result.

[0127] The feature map processing method in this embodiment can be understood as a layer containing one or more feature extractors (sometimes referred to as a dropout layer in this text). This feature extractor can be understood as a process or thread executed by a data processing device calling program code. This dropout layer can be added to the aforementioned traditional neural network. For example, as shown... Figure 2C As shown, this dropout processing layer can be added between hidden layer 1 131 and hidden layer 2 132 as a new hidden layer in the neural network layer 130. In this case, the first feature map, which will be described later, is the output of hidden layer 1 131 and the input of the dropout processing layer; the second feature map is the output of the dropout processing layer and the input of hidden layer 2 132. For example, as... Figure 2D As shown, this dropout processing layer can be added after the convolutional layer / pooling layer and before the hidden layer, for example, after layer 126 and before hidden layer 1131. Furthermore, this dropout processing layer can also replace a layer in a traditional neural network that has a similar dropout function. In this case, it can be understood that the feature map processing method provided in the embodiments of this application is abstracted into an operator, which can be added to any layer in a neural network or replace the operator originally used for dropout processing of the feature map in any layer. Assuming the aforementioned... Figure 2B Layers 124, 121, and 126 in the above structure have functions similar to discard processing, so layers 124, 121, and 126 can be replaced with the aforementioned discard processing layer to obtain the following result: Figure 2E The new neural network shown allows for more reasonable discarding of input feature maps or initial matrices while avoiding overfitting during neural network model training. It should be noted that... Figure 2C , Figure 2D as well as Figure 2EThis is merely an example of using a dropout processing layer to build a new neural network. In practical applications, this dropout processing layer can be combined with other layers in a traditional neural network as needed to improve the overall performance of the neural network. The location of the dropout processing layer within the neural network is not limited in this embodiment.

[0128] Based on the aforementioned system architecture and application scenarios, the following section combines... Figure 3 This application introduces a feature map processing method, the aforementioned Figure 1 The data processing device 001 in the middle will perform the following steps:

[0129] 301. Obtain the first feature map.

[0130] The first feature map is the input to the aforementioned feature extractor. When the aforementioned discarding layer contains only one feature extractor, the first feature map can be the input to the discarding layer. In one optional implementation, the first feature map is a feature map obtained by performing at least one convolution operation on an initial matrix. The aforementioned initial matrix can be a training sample, such as an image or video; or, for example, a matrix composed of data points converted from audio data. Furthermore, the aforementioned initial matrix can also be a feature map output from a neural network after processing by convolutional or pooling layers. In another optional implementation, the aforementioned initial matrix can be directly used as the first feature map. That is, the aforementioned image, video, and matrix composed of data points converted from audio data are directly input into the aforementioned discarding layer for discarding processing. Specific details are not limited here.

[0131] Furthermore, when the aforementioned training samples are images, the initial matrix can be a matrix composed of the pixel values ​​in the image.

[0132] For example, when the aforementioned first feature map is determined by training samples of an image class (i.e., the aforementioned initial matrix is ​​an image), the first feature map can be one of one or more feature maps determined by the aforementioned image. The aforementioned image can be a single-channel image, a three-channel image, or a four-channel image. A single-channel image, also known as a grayscale image, represents each pixel with a single pixel value. If the pixel value of a single-channel image is represented by 8 bits, the range of the pixel value is 0 (black) to 255 (white). A three-channel image often refers to an RGB image, which can represent both color and black-and-white images. If the pixel value of a three-channel image is represented by 8 bits, each pixel is represented by three values: red (0-255), green (0-255), and blue (0-255) superimposed. A four-channel image adds a luminance channel to the three-channel image to represent transparency. When the luminance channel pixel value is 0, it represents complete transparency; when the luminance channel pixel value is 255, it represents complete opacity.

[0133] Generally, when the training sample is a single-channel image, at least one feature map can be obtained by convolution operation on the single-channel image. The number of convolution kernels is equal to the number of feature maps in the single-channel image. When the training sample is a three-channel image, in order to preserve the features of different dimensions of the image as much as possible, convolution operation is performed on the matrix formed by the pixel values ​​of each channel. That is, the aforementioned three-channel image can obtain at least three feature maps by convolution operation, and each feature map corresponds to the matrix formed by the pixel values ​​of one channel in the aforementioned three-channel image. Similarly, when the training sample is a four-channel image, at least four feature maps can be obtained after convolution operation. In this embodiment, the first feature map can be the feature map obtained by convolution operation on the aforementioned single-channel image, or it can be one of the four feature maps obtained by convolution operation on the four-channel image; the specific choice is not limited here.

[0134] The following example uses single-channel images as training samples. Figure 4AAs shown, a single-channel image 401 contains white and black pixels, and the pixel value of each pixel in the single-channel image 401 constitutes a matrix 402. For example, the pixel value of a black pixel is 0, and the pixel value of a white pixel is 255. This matrix 402 reflects the distribution of each pixel in the single-channel image 401. After convolving this matrix 402 with a convolution kernel 403, a feature map 404 of the single-channel image is obtained. This feature map 404 is also a matrix containing multiple elements, each with a specific value. It should be understood that the features extracted from matrix 402 using different convolution kernels are not entirely the same. For example, convolution kernel 403 is a 3×3 convolution kernel, with only the top-left and bottom-right corners having a value of 1, and all other values ​​in convolution kernel 403 being 0. Thus, when convolution kernel 403 scans through matrix 402, it represents the extracted pixel values ​​of the top-left and bottom-right corners of the scanned area. Since the convolution kernel 403 is a 3×3 convolution kernel, the aforementioned scanning region can be any 3×3 matrix in matrix 402. In this example, matrix 402 is the aforementioned initial matrix, and feature map 404 is the aforementioned first feature map.

[0135] It should be understood that in practical applications, different pixels can also be represented by custom pixel values. For example, black pixels can be defined as 1, and white pixels as -1. Specific limitations are not specified here. Figure 4B As shown, still using the aforementioned single-channel image as an example, when defining black pixels as 1 and white pixels as -1, a matrix 405 composed of custom pixel values ​​can be obtained. This matrix 405 can reflect the distribution of black and white pixels in the single-channel image 401. After convolving this matrix 405 with a 5×5 convolution kernel 406, a feature map 407 of matrix 405 is obtained. In this example, matrix 405 is the aforementioned initial matrix, and feature map 407 is the aforementioned first feature map. It should be understood that in practical applications, the image to be processed contains tens of thousands of pixels with different pixel values. The example in this embodiment is only for the convenience of subsequent introduction and does not limit the number of different pixels in the image to be processed.

[0136] Furthermore, the aforementioned first feature map can also be a feature map output from a neural network after processing by convolutional or pooling layers. For example, Figure 4C As shown, the aforementioned Figure 4B Feature map 407 is then convolved again using kernel 408 to obtain feature map 409. In this example, feature map 407 is the aforementioned initial matrix, and feature map 409 is the aforementioned first feature map.

[0137] As can be seen from the above examples, the initial matrix in this embodiment can be any of the aforementioned implementation methods. For ease of subsequent explanation, only an image is used as an example for the initial matrix. In the following text, it will be stated that the initial matrix contains multiple pixels, and each pixel has a pixel value.

[0138] In this embodiment, the first feature map contains multiple elements, each corresponding to a region in the initial matrix, also known as the receptive field. The size of this region is the same as the size of the convolution kernel; that is, the size of the region is determined by the size of the convolution kernel. Figure 4D Let's take an example. When matrix 405 is the initial matrix, the first feature map is feature map 407. At this time, the element a0 with a value of "-1" in the upper left corner of feature map 407 is obtained by convolving region a1 in matrix 405 with convolution kernel 406. Therefore, element a0 in feature map 407 corresponds to region a1 in matrix 405. Similarly, the element b0 with a value of "-5" in the lower right corner of feature map 407 is obtained by convolving region b1 in matrix 405 with convolution kernel 407. Therefore, element b0 in feature map 407 corresponds to region b1 in matrix 405. And so on, each element in feature map 407 corresponds to a region in matrix 405.

[0139] Furthermore, the set of one or more elements in the first feature map can be understood as a feature, which is related to the initial matrix and the convolution kernel. When the initial matrix and / or the convolution kernel change, the first feature map will change, and the features reflected by the first feature map will be different.

[0140] When the initial matrix is ​​determined, the size of the extraction region (or scanning region) in the initial matrix can be adjusted by changing the size of the convolution kernel, thereby controlling the granularity of the features scanned from the initial matrix; the types of features extracted from the initial matrix can be adjusted by changing the values ​​in the convolution kernel. Therefore, controlling the convolution operation allows control over the types and granularity of features extracted from the initial matrix. This can also be understood as the first feature map reflecting the types and granularity of features extracted from the initial matrix.

[0141] Optionally, when the first feature map is obtained by performing multiple convolution operations on the initial matrix, each element in the first feature map can indirectly reflect the distribution characteristics of each pixel (or data point) in the initial matrix. In this case, one element in the first feature map is determined by the multiple convolution kernels used in the aforementioned multiple convolution operations.

[0142] Furthermore, since the initial matrices determined for different images to be processed are different, the first feature map determined after convolution will also be different. Therefore, to a certain extent, the first feature map can reflect the characteristics of the initial matrix; in other words, the initial matrix can influence the elements presented in the first feature map.

[0143] It should be understood that the aforementioned convolutional kernels are based on automatic machine learning. The more times the convolution operation is performed, the more accurately the learned convolutional kernel can extract features.

[0144] It should be noted that when determining the aforementioned first feature map, the convolution kernel can be adjusted based on the actual model training requirements. The first feature map obtained by any of the aforementioned implementation methods can reflect the features in the initial matrix (e.g., the image to be processed) to a certain extent. In subsequent embodiments, the aforementioned convolution operation will not be described in detail.

[0145] It should be understood that, in this embodiment, the first feature map can be derived from... Figure 1 The feature map obtained by the data processing device 001 from an external device can also be a feature map calculated by the data processing device 001 based on the initial matrix obtained from the client device 003 or the database 002. The specific details are not limited here.

[0146] 302. Determine the autocorrelation information based on the first feature map.

[0147] The autocorrelation information is used to indicate the degree of correlation between the elements in the first feature map. It can also be understood as the degree of dependence between the values ​​of the elements in the first feature map. Generally, if different elements reflect different features, the autocorrelation information determined based on two or more different elements indicates a lower correlation; if different elements reflect the same or similar features, the autocorrelation information determined based on two or more different elements indicates a higher correlation. Therefore, the data processing device can calculate the autocorrelation information between the elements in the first feature map to determine whether the features indicated by the elements distributed in the first feature map are similar or close. Furthermore, it can determine which elements in the first feature map should be discarded and which should be retained based on the aforementioned autocorrelation information.

[0148] In this embodiment, when the degree of correlation indicated by the autocorrelation information of two or more elements in the first feature map is greater, the dependency relationship between the two or more elements is stronger. Therefore, discarding or missing one element has a greater impact on the other element, and the same operation should be performed on the two or more elements during the calculation. For example, two or more elements with high correlation can be retained simultaneously, or two or more elements with high correlation can be discarded simultaneously. Conversely, when the degree of correlation indicated by the autocorrelation information of two or more elements in the first feature map is smaller, the dependency relationship between the two or more elements is weaker, and discarding or missing one element has a smaller impact on the other element. In practical applications, when the correlation between two elements is low, it may reflect that the element corresponds to a region on the initial matrix that records different features. In this case, different operations are performed on the two elements in the first feature map. For example, when the correlation between two elements is low, if one element is retained, the other element is discarded.

[0149] It should also be understood that, since each element in the first feature map comes from one or more pixels in the initial matrix, this autocorrelation information can also reflect the correlation between different regions in the initial matrix. In other words, the aforementioned autocorrelation information can reflect the dependency between different regions in the initial matrix.

[0150] Still with Figure 4D For example, the autocorrelation information between elements a0 and b0 in feature map 407 can reflect the degree of correlation or dependence between regions a1 and b1 in matrix 405. Therefore, the dropout mask determined based on this autocorrelation information can change with the elements in the feature map, or with the data points in the initial matrix, rather than being a fixed probability value. Thus, after processing a feature map with a dropout mask, some elements in the feature map can be adaptively discarded. Based on reducing the probability of overfitting during model training, features in the initial matrix can be selectively retained or discarded, so that the second feature map obtained after the discarding process can be better applied to model training than the first feature map (when the initial matrix is ​​image data, the second feature map obtained after the discarding process can be better applied to the model training corresponding to that image than the first feature map).

[0151] For ease of understanding, we will use an image containing a person, sky, and forest as an example to illustrate the concept. In the first feature map determined by the aforementioned initial matrix, the person region corresponds to multiple elements, denoted as ai (i = 1, 2, 3, ...); the sky region corresponds to multiple elements, denoted as bi (i = 1, 2, 3, ...); and the forest region corresponds to multiple elements, denoted as ci (i = 1, 2, 3, ...). If the autocorrelation information between the multiple elements ai corresponding to the person region indicates a high correlation, the same operation needs to be performed on all elements ai in the same person region. For example, all elements ai in the person region can be retained, or all elements ai in the person region can be discarded. Conversely, if the autocorrelation information between the elements ai in the person region and the elements bi in the sky region indicates a low correlation, the opposite operation needs to be performed on the elements ai in the person region and the elements bi in the sky region. For example, we can try to retain the element ai corresponding to the portrait area while discarding the element bi corresponding to the sky area, or we can try to discard the element ai corresponding to the portrait area while retaining the element bi corresponding to the sky area.

[0152] In this embodiment, the aforementioned autocorrelation information can be semantic-based autocorrelation information (e.g., semantic correlation) or spatial-based autocorrelation information (e.g., spatial correlation), and no specific limitation is made here.

[0153] In one optional implementation, the autocorrelation information includes a correlation coefficient matrix. This correlation coefficient matrix can be a spatial correlation coefficient matrix, used to indicate the degree of correlation between the spatial positions of different elements in the first feature map. The spatial correlation coefficient matrix is ​​used to indicate the degree of correlation or dependence between a subset of elements in the first feature map and another subset of elements in the first feature map.

[0154] For example, the transpose of the first feature map can be multiplied by the first feature map to obtain the spatial correlation coefficient matrix of the first feature map. For example, the aforementioned spatial correlation coefficient matrix can be calculated using the following formula 1-1:

[0155] correlation=(f(x)) T ·f(x)(Formula 1-1)

[0156] Where x is the initial matrix; f(x) is the first feature map obtained by convolution operation of the initial matrix; T is the matrix transpose operator, used to transpose the first feature map; and correlation is the spatial correlation coefficient matrix of the first feature map.

[0157] As mentioned above Figure 4BFeature map 407 in the example is the first feature map, which will be used for introduction. Figure 4E As shown, multiplying the transpose matrix 410 of feature map 407 with feature map 407 yields the correlation coefficient matrix 411. For example, when... Figure 5A When feature map 501 is the first feature map, multiplying the transpose matrix 502 of feature map 501 with feature map 501 yields the correlation coefficient matrix 503. For ease of explanation, the following text will refer to it as... Figure 5A The first feature map and correlation coefficient matrix shown are used as examples for introduction.

[0158] For example, the first feature map can be linearly transformed first, and then the transpose of the linearly transformed matrix can be multiplied by the linearly transformed matrix to obtain the aforementioned correlation coefficient matrix. For example, the aforementioned spatial correlation coefficient matrix can be calculated using the following formula 1-2:

[0159] correlation = A T (f(x))·B(f(x))(Formula 1-2)

[0160] Where x is the initial matrix; f(x) is the first feature map obtained by convolution operation on the initial matrix; A(*) and B(*) are convolution operators with a kernel size of 1×1, used to perform linear transformation on the first feature map; T is the matrix transpose operator, used to transpose the first feature map; and correlation is the spatial correlation coefficient matrix of the first feature map.

[0161] It should be understood that Formulas 1-1 and 1-2 described above are merely possible implementations for calculating the spatial correlation coefficient matrix in this embodiment, but are not limited to using only Formulas 1-1 and 1-2 when calculating the spatial correlation coefficient matrix. In other words, in practical applications, there may be other formulas for calculating the spatial correlation coefficient matrix, which are not limited in this embodiment.

[0162] It should be understood that the aforementioned spatial correlation coefficient matrix is ​​a specific implementation of autocorrelation information. When different formulas are used for calculation, the specific implementation of this autocorrelation information will differ. In other words, this autocorrelation information can also be represented by other matrices reflecting correlation; no specific limitation is made here. In this embodiment and subsequent embodiments, the spatial correlation coefficient matrix implementation will be used as an example for description.

[0163] 303. Determine the discard mask based on the autocorrelation information.

[0164] Specifically, the aforementioned spatial correlation coefficient matrix is ​​first normalized (or column-normalized) to obtain the first probability matrix. Then, the drop mask is determined based on the aforementioned first probability matrix.

[0165] The normalization process maps all elements in each row (or column) of the spatial correlation coefficient matrix to real numbers between 0 and 1. In one optional implementation, the sum of these real numbers is 1. Therefore, each element in the first probability matrix takes a value between 0 and 1. Optionally, the sum of all elements in each row (or column) of the first probability matrix is ​​1. This normalization process is equivalent to converting the spatial correlation coefficient matrix into probability or weight values ​​between 0 and 1; that is, each element in the first probability matrix represents a probability or weight value.

[0166] Furthermore, it should be understood that the aforementioned first probability matrix and the spatial correlation coefficient matrix are of the same size, and each element in the first probability matrix corresponds to the element at the same position in the spatial correlation coefficient matrix. Here, "same position" refers to the same row and column in the same matrix. For example, the element in the first row and first column of the first probability matrix corresponds to the element in the first row and first column of the spatial correlation coefficient matrix; the element in the first row and second column of the first probability matrix corresponds to the element in the first row and second column of the spatial correlation coefficient matrix; and so on. Moreover, the larger the value of an element in the spatial correlation coefficient matrix, the larger the value of the corresponding element in the first probability matrix.

[0167] For example, the aforementioned normalization process can employ the Softmax function. If the Softmax function is used to process the aforementioned spatial correlation coefficient matrix, the aforementioned first probability matrix can be calculated using the following formula 2-1:

[0168] probability=Softmax(correlation)(Formula 2-1)

[0169] Wherein, correlation is the spatial correlation coefficient matrix of the first feature map; probability is the normalized probability representation of correlation, i.e. the aforementioned first probability matrix; Softmax(*) is the flexible maximum value operator.

[0170] When processing the spatial correlation coefficient matrix using the Softmax function, the processing of each element in a row (or column) of the spatial correlation coefficient matrix can be represented by the following formula:

[0171]

[0172] Where ci is the i-th element of each row (or column) in the spatial correlation coefficient matrix, n is the total number of elements in each row (or column) in the spatial correlation coefficient matrix, and S(ci) is an element in the first probability matrix.

[0173] by Figure 5B For example, the correlation coefficient matrix 503 is normalized using the aforementioned formula 2 to obtain the probability matrix 504. In this example, the probability matrix 504 is the aforementioned first probability matrix. Taking the first row of the correlation coefficient matrix 503 as an example, the first element of the first row of the probability matrix 504 is: S(2)=e 2 / (e 2 +e 0 +e 0 S(0) ≈ 0.78; The second element of the first row of probability matrix 504 is: S(0) = e 0 / (e 2 +e 0 +e 0 )≈0.11; and so on, the specific details of this embodiment will not be repeated.

[0174] It should be understood that Formula 2-1 is only one way to achieve normalization as listed in this embodiment, but it is not limited to using Formula 2-1 only when normalizing the aforementioned spatial correlation coefficient matrix. That is to say, in practical applications, there may be other formulas to achieve normalization, which are not limited in this embodiment.

[0175] It should be understood that the aforementioned spatial correlation coefficient matrix is ​​only one specific implementation of autocorrelation information, and the spatial correlation coefficient matrix in step 303 can also be replaced by other matrices that reflect correlation.

[0176] Subsequently, the drop mask will be determined based on the aforementioned first probability matrix.

[0177] Specifically, a random number matrix of the same size as the first probability matrix is ​​obtained, where each element takes a value between 0 and 1. Then, this random number matrix is ​​binarized according to the first probability matrix; the result of this binarization is the discard mask. That is, each element of the first probability matrix is ​​used as a threshold for binarizing the corresponding element in the random number matrix. When an element in the random number matrix is ​​less than its corresponding element in the first probability matrix, the corresponding element in the discard mask is determined to have a first value; when an element in the random number matrix is ​​greater than its corresponding element in the first probability matrix, the corresponding element in the discard mask is determined to have a second value. The discard mask consists of these two different values. For example, 0 and 1 can be used to represent these two different values ​​respectively. For instance, when an element in the random number matrix is ​​less than its corresponding element in the first probability matrix, the corresponding element in the discard mask is determined to have a value of 0; when an element in the random number matrix is ​​greater than its corresponding element in the first probability matrix, the corresponding element in the discard mask is determined to have a value of 1. At this point, the discard mask is a matrix consisting of values ​​0 and 1.

[0178] In this embodiment, the threshold for generating the drop mask (i.e., the first probability matrix) is determined by the autocorrelation information of the first feature map, which reflects the correlation between each element in the first feature map. Therefore, as the elements in the first feature map change, the autocorrelation information will also change, leading to a change in the drop mask. Compared to the conventional approach of using a fixed value as the threshold for generating the drop mask, the threshold used in this embodiment can adaptively change, thus enabling adaptive adjustment of the drop mask.

[0179] It should be understood that, since the first probability matrix and the random number matrix are of the same size, and an element in the first matrix and an element in the random number matrix determine an element in the drop mask, the three matrices—the drop mask, the first probability matrix, and the random number matrix—are of the same size.

[0180] In the aforementioned process, the random number matrix is ​​binarized using a first probability matrix to obtain a discard mask. This discard mask carries not only the randomness of the random number matrix but also probabilities determined based on correlation information. When this discard mask is applied to the feature map, it introduces a relatively reasonable perturbation. This relatively reasonable perturbation considers not only randomness but also correlation. Therefore, it can discard some elements in the feature map while retaining as many correlated features as possible, resulting in the subsequent discard output containing features more conducive to model training.

[0181] For example, the aforementioned drop mask can be calculated using the following formula 3:

[0182]

[0183] Where P represents probability, which is the normalized probability representation of correlation, i.e. the first probability matrix mentioned above; R represents the random number matrix; m(x) is the drop mask, which can also be called the self-attention drop mask.

[0184] like Figure 5CAs shown, probability matrix 504 is obtained by normalizing correlation coefficient matrix 503. Random number matrix 505 consists of random numbers between 0 and 1 randomly generated by data processing device 001 using a random number generator. When the discard mask is determined using the aforementioned formula 3, the first element of the first row of random number matrix 505 is 0.65, which is less than the first element of the first row of probability matrix 504 (0.78), so the first element of the first row of the discard mask is 0; similarly, the second element of the first row of random number matrix 505 is 0.23, which is greater than the first element of the first row of probability matrix 504 (0.11), so the first element of the first row of the discard mask is 1. And so on, ultimately yielding... Figure 5C The discard mask is 506.

[0185] 304. Determine the second feature map based on the discard mask and the first feature map.

[0186] In this embodiment, the second feature map can be the output of the aforementioned feature extractor. When the aforementioned discard processing layer contains only one feature extractor, the second feature map can be the output of the discard processing layer. When the discard processing layer is connected to a next layer (e.g., a hidden layer, an output layer, etc.), the second feature map will be used as the input of the next layer.

[0187] The second feature map is the feature map obtained by applying the discard mask to the first feature map after a discard operation; that is, the remaining part of the first feature map after discarding some features. Therefore, the second feature map can also be called the discarded output. It should be noted that the size of the discard mask is the same as the size of the first feature map. After the discard operation, the size of the output second feature map is the same as the size of the aforementioned first feature map. For example, if the first feature map is a 9×9 matrix, then the discard mask should also be a 9×9 matrix, and the finally calculated second feature map should also be a 9×9 matrix.

[0188] In this embodiment, the discard mask is used to indicate elements to be discarded. "Discarding" in this embodiment can be understood as follows: when the discard mask is applied to a feature map, some elements in the feature map will be set to zero or modified. It can also be understood as changing the value of an element in the feature map from a non-zero value to 0, or changing the value of an element in the feature map to a non-zero value. Specifically, when the discard mask is applied to a feature map, the product of the discard mask and the aforementioned feature map is first calculated, and then the difference between the feature map and the aforementioned product is calculated. The product of the discard mask and the feature map is used to indicate the elements that should be discarded from the feature map and the degree of discarding.

[0189] It should be understood that the aforementioned process of applying the discard mask to the feature map can also be called the process of performing a discard operation on the feature map using the discard mask.

[0190] In this embodiment, the process of determining the second feature map based on the discard mask and the first feature map can be implemented in the following two optional ways:

[0191] In implementation method one, the discard mask is applied directly to the first feature map. Specifically, as follows... Figure 6A As shown, the product of the discard mask and the first feature map can be calculated first to obtain the third feature map; then, the difference between the first feature map and the third feature map can be calculated to obtain the second feature map.

[0192] In this embodiment, the discard mask is multiplied with the first feature map. When an element in the discard mask is 0, it means that the corresponding element in the first feature map is retained; when an element in the discard mask is 1, it means that the element in the first feature map is discarded or modified.

[0193] For example, the aforementioned second feature map can be calculated using the following formula 4-1:

[0194] f'(x)=f(x)-m(x)·f(x) (Formula 4-1)

[0195] Where f(x) is the first feature map; m(x) is the discard mask; and f'(x) is the second feature map, i.e., the discard output.

[0196] Specifically, with Figure 6B and Figure 6C For example, the discard mask 506 is multiplied by feature map 501 to obtain feature map 507. The difference between feature map 501 and feature map 507 is taken to obtain feature map 508. In this example, feature map 501 is the first feature map, feature map 507 is the third feature map, and feature map 508 is the second feature map (i.e., the discarded output).

[0197] In this implementation, the discard mask is directly used to perform a discard operation on the first feature map in order to discard or modify some elements in the first feature map.

[0198] In the second implementation, the discard mask is applied to the product of the first probability matrix and the aforementioned first feature map. Specifically, as follows... Figure 7A As shown, the product of the discard mask and the fourth feature map can be calculated first to obtain the fifth feature map. The fourth feature map is either the product of the first probability matrix and the first feature map, or the product of the first probability matrix and a linear or nonlinear transformation of the first feature map. Then, the difference between the first feature map and the fifth feature map is taken to obtain the second feature map.

[0199] For example, the aforementioned second feature map can be calculated using the following formula 4-2:

[0200] f'(x)=f(x)-m(x)·g(x) (Formula 4-2)

[0201] Where f(x) is the first feature map; m(x) is the discard mask; g(x) is the fourth feature map; m(x)·g(x) is the fifth feature map; and f'(x) is the second feature map, i.e., the discard output.

[0202] Where g(x) = probability·f(x), g(x) is a modulation representation based on the self-attention mechanism; probability is the normalized probability representation of correlation, i.e. the first probability matrix mentioned above.

[0203] Alternatively, g(x) = probability·transformation[f(x)], where g(x) is a modulation representation that integrates linear modulation and self-attention mechanisms. Here, transformation is the modulation operator used to linearly or nonlinearly modulate f(x). When linear modulation is used, this transformation can be a 1×1 convolution operator C(*); when nonlinear modulation is used, this transformation can be a nonlinear activation function, such as the Sigmoid function, the Tanh function, and other functions or mappings with nonlinear functions, without specific limitations here. The Sigmoid function is a common sigmoid function in biology, also known as an sigmoid growth curve. Because the Sigmoid function is monotonically increasing and its inverse function is also monotonically increasing, it is often used as a threshold function in neural networks, mapping variables input to the Sigmoid function to the range of 0 to 1, thus achieving a nonlinear transformation. Furthermore, the Tanh function is one of the hyperbolic functions, derived from the basic hyperbolic functions (i.e., hyperbolic sine and hyperbolic cosine). The Tanh function is monotonically increasing and its domain is between 0 and 1; therefore, variables input to the Tanh function can be mapped to the range of 0 to 1, thus enabling nonlinear transformations.

[0204] For example, if the Sigmoid function is used to modulate the aforementioned first feature map, the following formula can be used.

[0205] Equation 5-1 calculates the aforementioned fourth feature graph g(x):

[0206]

[0207] Here, f(x) is the first feature map.

[0208] For example, if the Tanh function is used to process the aforementioned spatial correlation coefficient matrix, the following formula can be used.

[0209] Equation 5-2 calculates the aforementioned fourth feature graph g(x):

[0210]

[0211] Here, f(x) is the first feature map.

[0212] Specifically, with Figure 7B , Figure 7C and Figure 7D For example, multiplying probability matrix 504 by feature map 501 yields feature map 509. Then, applying drop mask 506 to feature map 509 yields feature map 510. Finally, subtracting feature map 509 from feature map 510 yields feature map 511. In this example, feature map 501 is the first feature map, feature map 509 is the fourth feature map, feature map 510 is the fifth feature map, and feature map 511 is the second feature map (i.e., the discarded output).

[0213] In this implementation, since the fourth feature map is obtained by transforming the first feature map according to the first probability matrix, or by transforming the transformation matrix (e.g., a linear or nonlinear transformation matrix) of the first feature map according to the first probability matrix, the fourth feature map carries most of the features of the first feature map, but it is not entirely identical to the first feature map. Therefore, multiplying the discard mask by the fourth feature map is equivalent to perturbing the first feature map. The method of determining the second feature map (i.e., discarding the output) by subtracting the first feature map from the fifth feature map introduces a perturbation to the first feature map, allowing for the retention of features in the first feature map that are more beneficial for model training while discarding some elements.

[0214] In this embodiment, a first probability matrix is ​​used to binarize the random number matrix to obtain a discard mask. When this discard mask is applied to the feature map, it introduces a perturbation to the elements in the feature map. However, this perturbation is not the random perturbation used in traditional techniques, but rather a perturbation generated according to the features in the feature map. Therefore, it is possible to discard a portion of the elements in the feature map while preserving as many features as possible that are more beneficial to model training. In other words, the entire discarding process is based on the attributes or features of the feature map, rather than being completely random.

[0215] To facilitate understanding, the following example will be used as an example:

[0216] In this embodiment, when the feature map processing method is applied to a classification model, for example, to classify datasets CIFAR-10 and CIFAR-100, where CIFAR-10 has 10 classes and CIFAR-100 has 100 classes, the convolutional neural network model used is assumed to have three convolutional layers, one pooling layer, and one fully connected layer. Each convolutional layer has 96, 128, and 256 convolutional kernels, respectively, and each layer contains a convolution operation with a stride of 1 and a kernel size of 5×5. A max pooling operation with a stride of 2 and a kernel size of 3×3 is then connected after the aforementioned convolutional layers. Then, two fully connected layers containing 2048 hidden nodes are connected. As mentioned above, a dropout processing layer can be added after each convolutional layer or fully connected layer.

[0217] Table 1 shows a comparison of experimental data between the feature map processing method and the traditional discarding method used in this application embodiment when applied to the aforementioned classification model scenario:

[0218] Table 1

[0219]

[0220] As shown in Table 1 above, the feature map processing method in this embodiment achieves optimal classification results on both datasets, reaching classification accuracies of 87.14% and 57.64%, respectively. The feature map processing method in this embodiment can be used in various scenarios such as object detection and segmentation based on neural networks. Furthermore, the network structure of the neural network can be constructed according to scenario requirements. For example, after any layer that can output a feature map, the aforementioned discarding layer can be optionally connected. For details, please refer to the aforementioned... Figures 2A to 2E The neural network structure shown is not described in detail here.

[0221] like Figure 8 The diagram shown is a structural schematic of a data processing device 80 provided in an embodiment of this application. This data processing device 80 can be a server, analyzer, large-scale computing device, or large-scale management device; no specific limitation is made here. (The foregoing...) Figure 3 The steps in the corresponding embodiments can be executed by the data processing device 80.

[0222] The data processing apparatus 80 includes at least one processor 801 and at least one memory 802. The processor 801 and the memory 802 are interconnected via a circuit. It should be understood that... Figure 8 Only one processor 801 and one memory 802 are shown.

[0223] The processor 801 can be a general-purpose central processing unit (CPU), a microprocessor, a network processor (NP), or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of programs according to the present application. The processor 801 can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. The processor 801 can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Furthermore, the processor 801 can be a standalone semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it can form a system-on-a-chip (SoC) with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits), or it can be integrated as a built-in processor within an application-specific integrated circuit (ASIC). This ASIC with integrated processor can be packaged separately or together with other circuits.

[0224] Furthermore, the aforementioned memory 802 can be a read-only memory (ROM), or other types of static storage devices capable of storing static information and instructions, or random access memory (RAM), or other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM); the specific type is not limited here. The memory 802 can exist independently, but is connected to the aforementioned processor 801. Optionally, the memory 802 can also be integrated with the aforementioned processor 801. For example, it can be integrated into one or more chips.

[0225] Furthermore, the memory 802 is also used to store program code that executes the technical solutions of the embodiments of this application. The aforementioned program code can be controlled and executed by the processor 801, and various types of computer program code being executed can also be considered as drivers for the processor 801. Therefore, the processor 801 can determine a drop mask based on the autocorrelation information of the first feature map, and then determine a drop output based on the first feature map and the drop mask. Optionally, the processor 801 can perform a convolution operation on the initial matrix to determine the aforementioned first feature map.

[0226] Optionally, the data processing device 80 further includes a communication interface 803 for communicating with other servers or network devices. The data processing device 80 can receive instructions or data from other devices through the communication interface 803. For example, the communication interface 803 is a transceiver, such as a wireless transceiver or an optical transceiver.

[0227] For example, the data processing device 80 can receive a first feature map from a client device or a database through the communication interface 803. As another example, the data processing device 80 can receive an initial matrix (e.g., data to be processed or an image to be processed) from a client device through the communication interface 803. Furthermore, the data processing device 80 can also send instructions or data to other devices through the communication interface 803. For example, the data processing device 80 can send a drop mask and / or a second feature map determined based on the first feature map to a client device through the communication interface 803.

[0228] This application also provides a computer-readable storage medium storing a program for generating a drop mask and a program for processing a feature map based on the drop mask. When the aforementioned computer program is run on a computer, the computer can perform the aforementioned actions. Figure 3 The method described in the illustrated embodiment.

[0229] In one possible implementation, the aforementioned Figure 8 The data processing device 80 shown is a chip, also known as a digital processing chip. This chip includes a processing unit and a communication unit. The processing unit receives program instructions through the communication unit, and these program instructions are executed by the processing unit to cause it to perform the aforementioned actions. Figure 3 The method steps described in the corresponding embodiment are as follows. Specifically, the processing unit is a circuit that integrates the processor 801 or implements the functions of the processor 801, and the communication unit is a circuit or interface that integrates the communication interface 803 or implements the functions of the communication interface 803.

[0230] Optionally, when a storage unit is integrated into the chip, the storage unit can be a memory or other storage device. In this case, the processing unit in the chip can call program code from the storage unit to implement the aforementioned functionality. Figure 3 The method steps described in the corresponding embodiments are as follows. When the chip does not integrate a memory unit, the chip can be connected to an external memory or other storage device through the aforementioned communication unit, so as to obtain program code from the aforementioned external memory to implement the aforementioned... Figure 3 The method steps described in the corresponding embodiments.

[0231] For example, the aforementioned Figure 8 When the data processing device 80 shown is a chip, the specific interface of the chip can be as follows: Figure 9 As shown. Figure 9 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processing unit (NPU) 90, which is mounted as a coprocessor on the host CPU, where tasks are allocated. The core of the neural network processor 90 is the arithmetic circuit 901. The neural network processor 90, through a controller 902, controls the arithmetic circuit 901 to retrieve matrix data from memory (e.g., weight memory and input memory, described later) and perform multiplication operations.

[0232] The aforementioned arithmetic circuit 901 can be implemented in several ways. In one possible implementation, the arithmetic circuit 901 includes multiple processing engines (PEs). In another possible implementation, the arithmetic circuit 901 is a two-dimensional pulsating array. In yet another possible implementation, the arithmetic circuit 901 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In yet another possible implementation, the arithmetic circuit 901 is a general-purpose matrix processor. The specific embodiments of this application do not limit the specific form of the aforementioned arithmetic circuit 901.

[0233] For example, suppose the input memory 904 stores the input matrix A, and the weight memory 903 stores the weight matrix B. The arithmetic circuit 901 will retrieve the corresponding data of matrix B from the weight memory 903 and cache it in each PE of the arithmetic circuit 901. Furthermore, the arithmetic circuit 901 will also retrieve matrix A from the input memory 904, and then perform matrix operations using the aforementioned matrices A and B to obtain the output matrix C. This output matrix C can be a partial result or a final result, and it will be stored in the accumulator 905.

[0234] In addition, the neural network processor 90 includes a unified memory 906 for storing input and output data. Generally, input data can be moved to the unified memory 906 via a direct memory access controller (DMAC) 907. Some input or output data can be moved to the weight memory 903 via the DMAC 907. For example, externally input weight data (e.g., a weight matrix) can be moved to the aforementioned weight memory 903 via the DMAC 907.

[0235] Furthermore, the neural network processor 90's bus interface unit (BIU) 908 is used to interact with the memory access controller 907 and the instruction fetch buffer (IFB) 909 via an advanced extensible interface (AXI). In addition, the bus interface unit 908 is also used to fetch instructions from external memory via the instruction fetch buffer 909, and to fetch input data from external memory via the memory access controller 907, such as the original data of input matrix A or weight matrix B in the aforementioned example.

[0236] Optionally, the neural network processor 90 also includes a vector computation unit 910. This vector computation unit 910 includes multiple arithmetic processing units. When needed, the vector computation unit 910 can further process the output of the arithmetic circuit 901. For example, vector multiplication, vector addition, exponential operations, logarithmic operations, or size comparisons. Furthermore, the vector computation unit 910 is also used for computation in non-convolutional / fully connected layers of the neural network, such as batch normalization (also known as batch standardization), pixel-level summation, and upsampling of feature maps.

[0237] In one possible implementation, the aforementioned vector calculation unit 910 can store the processed output vector in a unified memory 906, so that the arithmetic circuit 901 can call the vectors, functions, and other data in the unified memory 906 during subsequent calculations. For example, the vector calculation unit 910 can store linear and / or nonlinear functions in the unified memory 906, so that the vector calculation unit can call the linear and / or nonlinear functions from the unified memory 906 to further process the output of the arithmetic circuit 901, and store the processing results in the unified memory 906 or the accumulator 905. For another example, the vector calculation unit 910 can perform linear or nonlinear transformations on the feature maps extracted by the convolutional layer; for yet another example, the vector calculation unit 910 can calculate the correlation of the feature maps. In another possible implementation, the vector calculation unit 910 can also be used to generate normalized values ​​(e.g., normalizing the correlation coefficient matrix to obtain a first probability matrix), pixel-level summation values ​​(e.g., using a convolutional kernel to extract features from the initial matrix), or both. In another possible implementation, the processed output vector can be used as an activation input to the arithmetic circuit 901 so that the output vector can be used in subsequent hidden layers in the neural network.

[0238] In addition, the aforementioned controller 902 is also connected to the instruction fetch memory 909 so that the instruction fetch memory 909 can store the instructions used by the controller 902.

[0239] It should be understood that the aforementioned weight memory 903, input memory 904, unified memory 906, and instruction fetch memory 909 are all on-chip memories. External memories are proprietary to the neural network processor hardware architecture.

[0240] The operations of each layer in the neural network can be performed by the arithmetic circuit 901 or the vector calculation unit 910.

[0241] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more processors used to control the above. Figure 3 The method is executed by an integrated circuit, and the specific method is not limited here.

[0242] like Figure 10 The diagram shown is a structural schematic of a data processing apparatus 1000 according to an embodiment of this application. The data processing apparatus 1000 operates the following modules to achieve the aforementioned... Figure 3 The function of the data processing device in the corresponding method embodiment.

[0243] The data processing device 1000 includes a processing module 1001, a transceiver module 1002, and a storage module 1003.

[0244] The processing module 1001 is used to implement the aforementioned Figure 8 The processor 801 in the transceiver module performs functions such as determining autocorrelation information based on the first feature map and determining the discard output corresponding to the first feature map based on the autocorrelation information. The transceiver module 1002 is used to implement the aforementioned... Figure 3 Corresponding to the transmission and reception functions in the corresponding and other embodiments, for example, acquiring a first feature map and a random number matrix from an external device; or, for example, acquiring first image data or first audio data from an external device. The storage module 1003 is used to implement the aforementioned... Figure 8 The first memory 802 in the memory has the following functions. For example, it stores the correlation coefficient matrix, the first probability matrix, and the discard mask in the aforementioned embodiments. It also stores intermediate processing results such as the third feature map, the fourth feature map, and the fifth feature map.

[0245] For example, the processing module 1001 is used to determine a drop mask based on the autocorrelation information, the drop mask being used to indicate the element to be dropped; and to determine a second feature map based on the drop mask and the first feature map, the second feature map being the drop output.

[0246] For example, the processing module 1001 is used to normalize the correlation coefficient matrix to obtain a first probability matrix; and to use the first probability matrix as a threshold for binarization to binarize the first random number matrix to obtain the discard mask.

[0247] For example, the processing module 1001 is used to perform a unidirectional operation on the two elements when the autocorrelation information indicates that the correlation between the two elements is greater than a first threshold. The unidirectional operation includes discarding the two elements or retaining the two elements.

[0248] For example, the processing module 1001 is used to perform a reverse operation on the two elements when the autocorrelation information indicates that the correlation between the two elements is less than the first threshold. The reverse operation includes discarding one of the two elements and keeping the other of the two elements, which are two related elements in the first feature map.

[0249] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. Components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. In practical applications, some or all of the modules / units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided in this application, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines.

[0250] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, and special-purpose components. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, portable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0251] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0252] In this article, descriptions such as "first," "second," etc., are used only to distinguish one object or operation from another, and do not necessarily require or imply any such actual relationship or order between these objects or operations.

[0253] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0254] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A feature map processing method, characterized in that, include: Obtain a first feature map corresponding to the first image data or the first audio data. The first feature map includes multiple elements and is used to indicate the distribution characteristics of the multiple elements. Determine the autocorrelation information of the first feature map, wherein the autocorrelation information is used to indicate the degree of correlation between the elements; The discard output corresponding to the first feature map is determined based on the autocorrelation information, and the discard output is the data of the first feature map after the discarding process. The step of determining the discard output corresponding to the first feature map based on the autocorrelation information includes: When the autocorrelation information indicates that the correlation between two elements is greater than a first threshold, a unidirectional operation is performed on the two elements, which includes discarding the two elements or retaining the two elements; or... When the autocorrelation information indicates that the correlation between the two elements is less than the first threshold, a reverse operation is performed on the two elements. The reverse operation includes discarding one of the two elements and keeping the other of the two elements, which are two related elements in the first feature map.

2. The method according to claim 1, characterized in that, The step of determining the discard output corresponding to the first feature map based on the autocorrelation information includes: A drop mask is determined based on the autocorrelation information, and the drop mask is used to indicate the element to be dropped; A second feature map is determined based on the discard mask and the first feature map, and the second feature map is the discard output.

3. The method according to claim 1 or 2, characterized in that, The autocorrelation information includes semantic-based autocorrelation information and / or spatial-based autocorrelation information.

4. The method according to claim 2, characterized in that, The autocorrelation information is a correlation coefficient matrix; Determining the drop mask based on the autocorrelation information includes: The correlation coefficient matrix is ​​normalized to obtain the first probability matrix; Obtain a first random number matrix, the size of which is the same as the size of the first probability matrix; Using the first probability matrix as a threshold for binarization, the first random number matrix is ​​binarized to obtain the discard mask.

5. The method according to claim 2 or 4, characterized in that, The discard mask is used to set some elements in the first feature map to zero or modify them.

6. The method according to claim 2 or 4, characterized in that, Determining the second feature map based on the drop mask and the first feature map includes: The third feature map is obtained by multiplying the discard mask and the first feature map; The second feature map is obtained by subtracting the first feature map from the third feature map.

7. The method according to claim 4, characterized in that, Determining the second feature map based on the drop mask and the first feature map includes: The fifth feature map is obtained by multiplying the discard mask and the fourth feature map. The fourth feature map is the product of the first probability matrix and the first feature map, or the fourth feature map is the product of the first probability matrix and the matrix obtained after linear or nonlinear transformation of the first feature map. The second feature map is obtained by subtracting the first feature map from the fifth feature map.

8. The method according to claim 4 or 7, characterized in that, The correlation coefficient matrix includes a spatial correlation coefficient matrix, which is used to indicate the degree of correlation between the positions of different elements in the first feature map in the spatial domain. Determining the autocorrelation information of the first feature map includes: Multiplying the transpose of the first feature map with the first feature map yields the spatial correlation coefficient matrix.

9. The method according to any one of claims 1 to 2, characterized in that, The first feature map is an initial matrix; or, the first feature map is obtained by performing at least one convolution operation on the initial matrix. Wherein, one of the elements in the first feature map corresponds to at least one pixel in the initial matrix, and the initial matrix includes a portion of the first image data or the first audio data.

10. The method according to any one of claims 2, 4, and 7, characterized in that, The dropout mask is located in the dropout processing layer of the neural network. The first feature map is the input of the dropout processing layer, the second feature map is the output of the dropout processing layer, the first feature map is the output of the layer above the dropout processing layer, and the second feature map is the input of the layer below the dropout processing layer.

11. The method according to claim 10, characterized in that, The upper layer is the input layer of the neural network; and / or, the lower layer is the output layer of the neural network.

12. The method according to claim 10, characterized in that, The upper layer and / or the lower layer are hidden layers in the neural network.

13. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire a first feature map corresponding to the first image data or the first audio data. The first feature map includes multiple elements and is used to indicate the distribution characteristics of the multiple elements. A calculation module is used to determine the autocorrelation information of the first feature map, wherein the autocorrelation information is used to indicate the degree of correlation between the elements. The calculation module is further configured to determine the discard output corresponding to the first feature map based on the autocorrelation information, wherein the discard output is the data of the first feature map after the discarding process; The computing module is specifically used for: When the autocorrelation information indicates that the correlation between two elements is greater than a first threshold, a unidirectional operation is performed on the two elements, which includes discarding the two elements or retaining the two elements; or... When the autocorrelation information indicates that the correlation between the two elements is less than the first threshold, a reverse operation is performed on the two elements. The reverse operation includes discarding one of the two elements and keeping the other of the two elements, which are two related elements in the first feature map.

14. The data processing apparatus according to claim 13, characterized in that, The computing module is specifically used for: A drop mask is determined based on the autocorrelation information, and the drop mask is used to indicate the element to be dropped; A second feature map is determined based on the discard mask and the first feature map, and the second feature map is the discard output.

15. The data processing apparatus according to claim 14, characterized in that, The autocorrelation information is a correlation coefficient matrix; The calculation module is specifically used to normalize the correlation coefficient matrix to obtain a first probability matrix; The acquisition module is further configured to acquire a first random number matrix, wherein the size of the first random number matrix is ​​the same as the size of the first probability matrix; The calculation module is specifically used to use the first probability matrix as a threshold for binarization processing to binarize the first random number matrix and obtain the discard mask.

16. The data processing apparatus according to claim 14 or 15, characterized in that, The computing module is specifically used for: The third feature map is obtained by multiplying the discard mask and the first feature map; The second feature map is obtained by subtracting the first feature map from the third feature map.

17. The data processing apparatus according to claim 15, characterized in that, The computing module is specifically used for: The fifth feature map is obtained by multiplying the discard mask and the fourth feature map. The fourth feature map is the product of the first probability matrix and the first feature map, or the fourth feature map is the product of the first probability matrix and the matrix obtained after linear or nonlinear transformation of the first feature map. The second feature map is obtained by subtracting the first feature map from the fifth feature map.

18. The data processing apparatus according to any one of claims 13 to 15, characterized in that, The correlation coefficient matrix includes a spatial correlation coefficient matrix, which is used to indicate the degree of correlation between the positions of different elements in the first feature map in the spatial domain. The computing module is specifically used for: Multiplying the transpose of the first feature map with the first feature map yields the spatial correlation coefficient matrix.

19. A data processing apparatus, characterized in that, The device includes a processor coupled to a memory storing a program, wherein when the program instructions stored in the memory are executed by the processor, the data processing apparatus causes the data processing apparatus to perform the method of any one of claims 1 to 12.

20. A computer-readable storage medium comprising a program that, when executed by a processing unit, performs the method as described in any one of claims 1 to 12.

21. A chip, characterized in that, The device includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. When the program instructions are executed by the processing unit, the chip implements the method of any one of claims 1 to 12.

Images