Data processing method, device and equipment applied to convolutional neural network

Abstract

The application provides a data processing method, device and equipment applied to a convolutional neural network, the method comprising: performing convolution calculation on received input feature data of a current layer to obtain initial output feature data of the current layer; inputting the input feature data to a channel attention structure to make the channel attention structure perform predetermined processing on the input feature data, output a channel correlation matrix, and multiply the initial output feature data and the channel correlation matrix to obtain final output feature data of the current layer. In the method, the channel attention structure is arranged in at least one layer of the convolutional neural network, and the obtained channel correlation matrix is used to calibrate spatial features. Since the channel correlation matrix can represent the similarity degree of transformation results of channels of the current layer under different shared convolution kernels, the importance of the channels can be accurately measured, and thus the spatial features can be accurately calibrated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision system recognition, and particularly relates to a data processing method and device applied to a convolutional neural network and equipment. BACKGROUND

[0002] A convolutional neural network (CNN) is a kind of neural network containing convolution calculation and having a deep structure. It has shown great performance gain in visual tasks, so the enhancement of its feature representation has always been the focus of research. Among them, channel attention is one of the representative works, including feature aggregation and descriptor transformation.

[0003] At present, global average pooling, max pooling, second-order pooling or axis direction separation pooling are usually used as spatial feature aggregation, and serial fully connected layers, row convolution or one-dimensional convolution are used as descriptor transformation.

[0004] However, the above method only models the channel importance through information integration (i.e. pooling in the spatial dimension and weighted sum in the channel dimension), which is not accurate enough in measuring the channel importance, so it cannot accurately calibrate the spatial features. SUMMARY

[0005] The present application provides a data processing method and device applied to a convolutional neural network, to obtain accurate feature data.

[0006] In a first aspect, the present application provides a data processing method applied to a convolutional neural network, at least one layer of the convolutional neural network being provided with a channel attention structure; the data processing method comprising:

[0007] receiving input feature data of a current layer, and performing convolution calculation on the input feature data to obtain initial output feature data of the current layer;

[0008] if the current layer is provided with a channel attention structure, inputting the input feature data to the channel attention structure to obtain a channel correlation matrix; wherein the channel attention structure outputs the channel correlation matrix by performing a predetermined processing on the input feature data; wherein the channel correlation matrix represents the similarity degree of the transformation results of the channels of the current layer under different shared convolution kernels;

[0009] multiplying the initial output feature data and the channel correlation matrix to obtain final output feature data of the current layer.

[0010] Further, the method described above, the channel attention structure is used to perform the following processing:

[0011] for each shared convolution kernel, calculating a convolution result of the shared convolution kernel and input feature data of each channel in the current layer;

[0012] performing dot product calculation on the convolution results corresponding to different shared convolution kernels to obtain the channel correlation matrix.

[0013] Further, the method as described above, after the calculation of the convolution result of the shared convolution kernel and the input feature data of each channel in the current layer for each shared convolution kernel, further comprises:

[0014] for each shared convolution kernel, calculating a convolution result of the shared convolution kernel and input feature data of each channel in the current layer;

[0015] respectively unfolding the feature map groups corresponding to different shared convolution kernels in the spatial dimension to obtain intermediate results corresponding to different shared convolution kernels;

[0016] the dot product calculation on the intermediate results corresponding to different shared convolution kernels to obtain the channel correlation matrix comprises:

[0017] performing dot product calculation on the intermediate results corresponding to different shared convolution kernels to obtain dot product calculation results, and folding the dot product calculation results in the spatial dimension to obtain the channel correlation matrix.

[0018] Further, the method as described above, the number of shared convolution kernels is three; the dot product calculation on the intermediate results corresponding to different shared convolution kernels to obtain dot product calculation results comprises:

[0019] determining a first shared convolution kernel and a second shared convolution kernel in the different shared convolution kernels;

[0020] calculating the dot product of the intermediate result corresponding to the first shared convolution kernel and the intermediate result corresponding to the second shared convolution kernel to obtain a first dot product result;

[0021] processing the first dot product result based on a Softmax function to obtain a correlation coefficient;

[0022] calculating the dot product of the correlation coefficient and the intermediate result corresponding to the third shared convolution kernel to obtain a second dot product result;

[0023] processing the second dot product result based on a Sigmoid function to obtain the dot product calculation result.

[0024] Further, the method as described above, before multiplying the initial output feature data and the channel correlation matrix, further comprises:

[0025] According to the matrix size corresponding to the initial output feature data, the channel correlation matrix is up-sampled; the up-sampled channel correlation matrix is consistent in size with the initial output feature data.

[0026] Further, in the method, a step size of the shared convolution kernel is not greater than a width of the input feature data; and the width of the input feature data is less than or equal to a length of the input feature data.

[0027] Further, in the method, a depth of the shared convolution kernel is not greater than a depth of the input feature data.

[0028] Further, in the method, a dimension of a convolution kernel corresponding to a layer provided with the channel attention structure in the convolutional neural network is greater than 1.

[0029] In a second aspect, an embodiment of the present application provides a data processing apparatus applied to a convolutional neural network, at least one layer in the convolutional neural network being provided with a channel attention structure; the data processing apparatus comprises:

[0030] a calculation module, configured to receive input feature data of a current layer, and perform convolution calculation on the input feature data to obtain initial output feature data of the current layer;

[0031] an input module, configured to input the input feature data to the channel attention structure to obtain a channel correlation matrix if the current layer is provided with the channel attention structure; the channel attention structure outputs the channel correlation matrix by performing predetermined processing on the input feature data; the channel correlation matrix represents a similarity degree of transformation results of channels of the current layer under different shared convolution kernels;

[0032] a processing module, configured to take a multiplication result of the initial output feature data and the channel correlation matrix as final output feature data of the current layer.

[0033] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor.

[0034] the memory is configured to store program instructions executable by the processor;

[0035] The processor is configured to invoke the program instructions in the memory to execute the data processing method applied to the convolutional neural network.

[0036] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing computer execution instructions, the computer execution instructions being executed by a processor to implement the data processing method applied to the convolutional neural network.

[0037] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the data processing method applied to a convolutional neural network according to the first aspect.

[0038] The present application provides a data processing method, device and equipment applied to a convolutional neural network. The input feature data of a current layer received is subjected to convolution calculation to obtain initial output feature data of the current layer. The input feature data is input to a channel attention structure to make the channel attention structure perform predetermined processing on the input feature data, output a channel correlation matrix, and multiply the initial output feature data by the channel correlation matrix as final output feature data of the current layer. That is, the present application sets a channel attention structure in at least one layer of the convolutional neural network, obtains a channel correlation matrix, and multiplies the initial output feature data by the channel correlation matrix, that is, uses the channel correlation matrix to calibrate spatial features. Since the channel correlation matrix can represent the similarity degree of the transformation results of the channels of the current layer under different shared convolution kernels, the importance of the channels can be accurately measured, so that the spatial features can be accurately calibrated. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0040] Figure 1 A flowchart of the data processing method applied to the convolutional neural network provided by an embodiment of the present application;

[0041] Figure 2 A flowchart of the data processing method applied to the convolutional neural network provided by an embodiment of the present application;

[0042] Figure 3 A schematic diagram of the channel attention structure provided by an embodiment of the present application;

[0043] Figure 4 A flowchart of the data processing method applied to the convolutional neural network provided by an embodiment of the present application;

[0044] Figure 5 A structural schematic diagram of the data processing device applied to the convolutional neural network provided by an embodiment of the present application;

[0045] Figure 6 A structural schematic diagram of the electronic device of the present application.

[0046] The specific embodiments of the application have been shown by the above drawings, and will be described in more detail hereinafter. These drawings and detailed description are not intended to limit the scope of the concept of the application in any way, but to illustrate the concept of the application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0047] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The same numbers are used in different drawings to represent the same or similar elements. The embodiments described in the following exemplary embodiments are not meant to represent all embodiments consistent with the application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the application.

[0048] Convolutional neural networks have shown great performance gains in visual tasks, and researchers have been exploring the enhancement of their feature representations. Among them, channel attention is one of the representative works, which refers to taking the aggregation of spatial dimension (HxW) features as channel descriptors, and transforming these descriptors as channel responses, which are considered as channel importance measures, and used to calibrate spatial features.

[0049] Currently, global average pooling, max pooling, second-order pooling or axis direction separation pooling are usually used as spatial feature aggregation, and serial fully connected layers, row convolution or one-dimensional convolution are used as descriptor transformation.

[0050] However, the above methods only model the channel importance through information integration (i.e. pooling in the spatial dimension and weighting in the channel dimension), ignoring the correlation between channels, so the measurement of channel importance is not accurate enough, and thus the spatial features cannot be accurately calibrated.

[0051] The data processing method, device and equipment applied to the convolutional neural network provided by the application aim to solve the above technical problems of the prior art.

[0052] The technical solutions of the application and how the technical solutions of the application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the application will be described below with reference to the drawings.

[0053] Embodiment one

[0054] Figure 1 The flowchart of the data processing method applied to the convolutional neural network provided by the embodiments of the application is shown in Figure 1 The data processing method applied to the convolutional neural network provided by the embodiments of the application includes the following steps:

[0055] Step 101, receiving input feature data of a current layer, and performing convolution calculation on the input feature data to obtain initial output feature data of the current layer.

[0056] Step 102, if the current layer is provided with a channel attention structure, inputting the input feature data to the channel attention structure to obtain a channel correlation matrix; wherein the channel attention structure outputs the channel correlation matrix by performing a predetermined processing on the input feature data; wherein the channel correlation matrix represents the similarity degree of the transformation results of the channels of the current layer under different shared convolution kernels.

[0057] Step 103, multiplying the initial output feature data and the channel correlation matrix to obtain the final output feature data of the current layer.

[0058] It should be noted that the execution subject of the data processing method applied to the convolutional neural network provided in the embodiment can be a data processing device applied to the convolutional neural network. In actual application, the data processing device applied to the convolutional neural network can be realized by a computer program, such as application software, a computer program, etc., or by a medium storing the related computer program, such as a U disk, an optical disk, etc., or by an entity device integrated or installed with the related computer program, such as a chip, etc.

[0059] In the embodiment, in order to accurately calibrate the spatial features, the correlation between channels needs to be considered, and therefore a channel attention structure can be provided in at least one layer of the convolutional neural network. The channel attention structure is a branch structure in the convolutional neural network, that is, an additional feature map transformation is introduced in addition to the backbone convolutional network, and the channels that maintain a high degree of similarity under different transformations are selected for enhancement.

[0060] Specifically, when performing data processing, the data processing device can receive input feature data of a current layer, and perform convolution calculation on the input feature data to obtain initial output feature data of the current layer, that is, spatial feature data to be calibrated. The input feature data is the feature data output by each channel of the previous layer.

[0061] At the same time, if the current layer is provided with a channel attention structure, the data processing device can input the input feature data to the channel attention structure to obtain a channel correlation matrix. Specifically, the channel attention structure can output the channel correlation matrix by processing the input feature data, and the channel correlation matrix can represent the similarity degree of the transformation results of the channels of the current layer under different shared convolution kernels.

[0062] The shared convolution kernel can be trained by a conventional method, such as stochastic gradient descent or other suitable training method, and the embodiment is not limited in this regard. In addition, the shared convolution kernel can be used by each channel of the current layer.

[0063] In addition, the channels of the current layer can include all channels of the current layer, or can include part of the channels of the current layer. Specifically, the part of the channels can be one or more channels that need to calculate the channel correlation.

[0064] Since the channels with higher similarity degree of the transformed results under different shared convolution kernels retain stronger specific information, enhancing such channels is conducive to the convergence and performance improvement of the network structure. Therefore, the data processing apparatus can take the multiplication result of the initial output feature data and the channel correlation matrix as the final output feature data of the current layer. Through multiplication, the channels retaining stronger specific information can be enhanced, thereby realizing accurate calibration of the spatial features.

[0065] In one example, the channels of the current layer that need to calculate the channel correlation are the A channel and the B channel, where the A channel has a higher similarity degree of the transformed results under different shared convolution kernels, that is, the value corresponding to the A channel in the channel correlation matrix is greater than the value corresponding to the B channel. Therefore, multiplying the initial output feature data and the channel correlation matrix can realize enhancement of the A channel, and further realize accurate calibration of the spatial features.

[0066] The data processing method applied to the convolutional neural network provided in the embodiment is configured to perform convolution calculation on the received input feature data of the current layer to obtain initial output feature data of the current layer. The input feature data is input to the channel attention structure to enable the channel attention structure to perform predetermined processing on the input feature data, output a channel correlation matrix, and take the multiplication result of the initial output feature data and the channel correlation matrix as the final output feature data of the current layer. That is, in the embodiment, the channel attention structure is arranged in at least one layer of the convolutional neural network to obtain the channel correlation matrix, and the initial output feature data is multiplied by the channel correlation matrix, that is, the spatial features are calibrated by using the channel correlation matrix. Since the channel correlation matrix can represent the similarity degree of the transformed results of the channels of the current layer under different shared convolution kernels, the importance of the channels can be accurately measured, and thus the spatial features can be accurately calibrated.

[0067] Embodiment Two

[0068] On the basis of the above embodiment one, in order to further illustrate the data processing method applied to the convolutional neural network provided in the embodiments of the present application, in step 102, the channel attention structure is used to perform the following processing: for each shared convolution kernel, the convolution result of the shared convolution kernel and the input feature data of the current layer channel is calculated; the convolution results corresponding to different shared convolution kernels are dot product calculated to obtain the channel correlation matrix.

[0069] In one example, the channels of the current layer that need to calculate the channel correlation are A channel and B channel, then for each shared convolution kernel, the convolution result of the shared convolution kernel and the input feature data of the A channel and the B channel is calculated, and the convolution results corresponding to different shared convolution kernels are dot product calculated. Wherein, if the statistical value of the row where the A channel is located in the dot product result is greater than the statistical value of the row where the B channel is located, it indicates that in the obtained channel correlation matrix, the value corresponding to the A channel is greater than the value corresponding to the B channel, that is, the A channel has a higher degree of similarity (i.e. channel correlation) of the transformation result under different shared convolution kernels than the B channel.

[0070] Figure 2 The flowchart of the data processing method applied to the convolutional neural network provided in the embodiments of the present application is shown in Figure 2 The data processing method applied to the convolutional neural network provided in the embodiments of the present application, on the basis of the above embodiment two, after calculating the convolution result of each shared convolution kernel and the input feature data of each channel in the current layer, further includes:

[0071] Step 201, for each shared convolution kernel, the convolution result of the shared convolution kernel and the input feature data of the current layer channel is calculated, and the feature map group corresponding to different shared convolution kernels is obtained.

[0072] Step 202, respectively unfolding the feature map group corresponding to different shared convolution kernels in the spatial dimension, and obtaining the intermediate result corresponding to different shared convolution kernels.

[0073] Correspondingly, the dot product calculation of the convolution results corresponding to different shared convolution kernels is performed to obtain the channel correlation matrix, including:

[0074] Step 203, the dot product calculation of the intermediate results corresponding to different shared convolution kernels is performed to obtain the dot product calculation result, and the dot product calculation result is folded in the spatial dimension to obtain the channel correlation matrix.

[0075] In this embodiment, Figure 3 The schematic diagram of the channel attention structure provided in the embodiments of the present application is shown in Figure 3As shown, in order to obtain the correlation matrix, that is, the degree of similarity of the results of the transformation of the channels of the current layer under different shared convolution kernels, the data processing apparatus can calculate, for each shared convolution kernel, the convolution result of the shared convolution kernel and the input feature data of the current layer channel, to obtain a feature map group corresponding to different shared convolution kernels.

[0076] wherein the number of shared convolution kernels is three, and the shape of the input feature data is C'xHxW, wherein C' is the depth, H is the length, and W is the width, that is, Figure 3 the leftmost cube in FIG. 6, the step of convolution is 2, and accordingly, the number of obtained feature map groups is three, and the shape of each feature map group is C'xH / 2xW / 2, that is, Figure 3 the three small cubes to the right of the leftmost cube in FIG. 6.

[0077] Next, the data processing apparatus can respectively expand the feature map groups corresponding to different shared convolution kernels in the spatial dimension according to a preset data expansion strategy, to obtain intermediate results corresponding to different shared convolution kernels, that is, Figure 3 the three bar-shaped bodies in the middle, each of which has a shape of C'xHW / 4.

[0078] wherein the intermediate result corresponding to each shared convolution kernel includes the convolution calculation result data corresponding to different channels under the shared convolution kernel.

[0079] Finally, the data processing apparatus can perform dot product calculation on the intermediate results corresponding to different shared convolution kernels to obtain dot product calculation results, and fold the dot product calculation results in the spatial dimension according to the data expansion strategy, to obtain the correlation matrix, that is, Figure 3 the second cube to the right in FIG. 6, which has a shape of C'xH / 2xW / 2.

[0080] In addition, Figure 3 as shown in FIG. 6, the lowermost part shows that the input feature data of the current layer is subjected to convolution calculation to obtain the initial output feature data of the current layer, and the initial output feature data is multiplied by the correlation matrix, so that the final output feature data of the current layer can be obtained, that is, Figure 3 the rightmost cube in FIG. 6, which has a shape of C'xHxW.

[0081] The data processing method applied to the convolutional neural network provided in the embodiment calculates, for each shared convolution kernel, the convolution result of the shared convolution kernel and the input feature data of each channel in the current layer, and then performs dot product calculation on the convolution results corresponding to different shared convolution kernels, so that the channel correlation matrix can be obtained, thereby laying a foundation for accurate spatial feature calibration.

[0082] Embodiment Three

[0083] Figure 4A flowchart of the data processing method applied to the convolutional neural network provided in the embodiments of the present application is shown in FIG. 3. Figure 4 As shown in FIG. 3, based on the second embodiment, the number of shared convolution kernels is three. In step 203, the data processing device performs dot product calculation on the intermediate results corresponding to different shared convolution kernels to obtain a dot product calculation result, including the following steps.

[0084] In step 301, the data processing device determines a first shared convolution kernel and a second shared convolution kernel in the different shared convolution kernels.

[0085] In step 302, the data processing device calculates the dot product of the intermediate result corresponding to the first shared convolution kernel and the intermediate result corresponding to the second shared convolution kernel to obtain a first dot product result.

[0086] In step 303, the data processing device processes the first dot product result based on a Softmax function to obtain a correlation coefficient.

[0087] In step 304, the data processing device calculates the dot product of the correlation coefficient and the intermediate result corresponding to a third shared convolution kernel to obtain a second dot product result.

[0088] In step 305, the data processing device processes the second dot product result based on a Sigmoid function to obtain the dot product calculation result.

[0089] In this embodiment, in order to obtain the dot product calculation result, the data processing device can first determine a first shared convolution kernel and a second shared convolution kernel in the different shared convolution kernels, and calculate the dot product of the intermediate result corresponding to the first shared convolution kernel and the intermediate result corresponding to the second shared convolution kernel to obtain a first dot product result, which is shown in the following formula:

[0090]

[0091] wherein φ i，j is the first dot product result, d i，l is the traversal of the convolution calculation result data corresponding to the i-th channel under the first shared convolution kernel, d' j，l is the traversal of the convolution calculation result data corresponding to the j-th channel under the second shared convolution kernel, i is the channel under the first shared convolution kernel, j is the channel under the second shared convolution kernel, and l is the position of the convolution calculation result data corresponding to each channel under each shared convolution kernel.

[0092] Next, the data processing device can process the first dot product result based on a Softmax function to obtain a correlation coefficient, and calculate the dot product of the correlation coefficient and the intermediate result corresponding to a third shared convolution kernel to obtain a second dot product result. The third shared convolution kernel is a shared convolution kernel other than the first shared convolution kernel and the second shared convolution kernel.

[0093] Then, the data processing apparatus can process the second dot product result based on a Sigmoid function, and obtain a dot product calculation result.

[0094] The data processing method applied to the convolutional neural network provided in the embodiment adopts the channel attention structure to obtain the correlation matrix, that is, the similarity degree of the transformation results of each channel of the current layer under different shared convolution kernels, and lays a foundation for accurate spatial feature calibration.

[0095] Embodiment Four

[0096] On the basis of the above-mentioned embodiment one, in an optional implementation, the dimension of the convolution kernel corresponding to the layer provided with the channel attention structure in the convolutional neural network, that is, the convolution kernel used for convolution calculation on the input feature data of the layer, is greater than 1, so as to improve the effect of convolution calculation and save the calculation resources.

[0097] Embodiment Five

[0098] On the basis of the above-mentioned embodiment one, in an optional implementation, before multiplying the initial output feature data and the channel correlation matrix, the method further includes: performing up-sampling on the channel correlation matrix according to the matrix size corresponding to the initial output feature data. The size of the up-sampled channel correlation matrix is consistent with that of the initial output feature data, so as to facilitate multiplication and ensure the accuracy of the result.

[0099] Embodiment Six

[0100] On the basis of the above-mentioned embodiment one, in an optional implementation, the step length of the shared convolution kernel is not greater than the width of the input feature data. The width of the input feature data is less than or equal to the length of the input feature data.

[0101] In another optional implementation, the depth of the shared convolution kernel is not greater than the depth of the input feature data.

[0102] In the above manner, the accuracy of the calculation result can be effectively ensured.

[0103] Embodiment Seven

[0104] Figure 5 The structural schematic diagram of the data processing apparatus applied to the convolutional neural network provided in the embodiment of the present application is as shown in Figure 5As shown, the data processing apparatus applied to the convolutional neural network provided by the embodiment comprises a calculation module 41, an input module 42 and a processing module 43. The calculation module 41 is configured to receive input feature data of a current layer and perform convolution calculation on the input feature data to obtain initial output feature data of the current layer. The input module 42 is configured to input the input feature data to a channel attention structure of the current layer to obtain a channel correlation matrix, if the current layer is provided with the channel attention structure. The channel attention structure outputs the channel correlation matrix by performing a predetermined processing on the input feature data. The channel correlation matrix represents a similarity degree of transformation results of channels of the current layer under different shared convolution kernels. The processing module 43 is configured to multiply the initial output feature data and the channel correlation matrix to obtain final output feature data of the current layer.

[0105] The data processing apparatus applied to the convolutional neural network provided by the embodiment performs convolution calculation on the received input feature data of the current layer to obtain initial output feature data of the current layer. The input feature data is input to the channel attention structure to make the channel attention structure perform a predetermined processing on the input feature data to output a channel correlation matrix. The multiplication result of the initial output feature data and the channel correlation matrix is taken as the final output feature data of the current layer. That is, in the embodiment, the channel attention structure is arranged in at least one layer of the convolutional neural network to obtain the channel correlation matrix. The initial output feature data is multiplied by the channel correlation matrix, that is, the spatial feature is calibrated by using the channel correlation matrix. Since the channel correlation matrix can represent the similarity degree of transformation results of channels of the current layer under different shared convolution kernels, the channel importance can be accurately measured, so that the spatial feature can be accurately calibrated.

[0106] In an optional implementation, the processing module 43 is further configured to calculate, for each shared convolution kernel, a convolution result of the shared convolution kernel and the input feature data of the channel of the current layer, perform dot product calculation on the convolution results corresponding to different shared convolution kernels to obtain the channel correlation matrix.

[0107] In an optional implementation, the processing module 43 is further configured to calculate, for each shared convolution kernel, a convolution result of the shared convolution kernel and the input feature data of the channel of the current layer to obtain a feature map group corresponding to different shared convolution kernels. The feature map group corresponding to different shared convolution kernels is respectively unfolded in a spatial dimension to obtain intermediate results corresponding to different shared convolution kernels. Dot product calculation is performed on the intermediate results corresponding to different shared convolution kernels to obtain a dot product calculation result. The dot product calculation result is folded in the spatial dimension to obtain the correlation matrix.

[0108] In an optional implementation, the number of shared convolution kernels is three, and the processing module 43 is further configured to determine a first shared convolution kernel and a second shared convolution kernel from the different shared convolution kernels; calculate a dot product of an intermediate result corresponding to the first shared convolution kernel and an intermediate result corresponding to the second shared convolution kernel to obtain a first dot product result; process the first dot product result based on a Softmax function to obtain a correlation coefficient; calculate a dot product of the correlation coefficient and an intermediate result corresponding to a third shared convolution kernel to obtain a second dot product result; and process the second dot product result based on a Sigmoid function to obtain the dot product calculation result.

[0109] In an optional implementation, the data processing apparatus applied to the convolutional neural network further includes a sampling module 44 configured to up-sample the channel correlation matrix according to a matrix size corresponding to the initial output feature data; and the up-sampled channel correlation matrix has a same size as the initial output feature data.

[0110] In an optional implementation, a step of the shared convolution kernel is not greater than a width of the input feature data; and the width of the input feature data is less than or equal to a length of the input feature data.

[0111] In an optional implementation, a depth of the shared convolution kernel is not greater than a depth of the input feature data.

[0112] In an optional implementation, a dimension of a convolution kernel corresponding to a layer provided with a channel attention structure in the convolutional neural network is greater than 1.

[0113] It should be noted that the technical solutions and effects of the data processing apparatus applied to the convolutional neural network provided in this embodiment can be referred to the related content of the foregoing method embodiments, and will not be described herein again.

[0114] Embodiment Eight

[0115] Figure 6 A structural schematic diagram of an electronic device of the present application is shown in FIG. 1, and the present application further provides an electronic device 500, which includes a memory 501 and a processor 502. Figure 6

[0116] The memory 501 is configured to store a program. Specifically, the program can include program code including computer execution instructions. The memory 501 can include a high-speed RAM memory and can also include a non-volatile memory such as at least one disk memory.

[0117] The processor 502 is configured to execute the program stored in the memory 501.

[0118] ​The computer program is stored in the memory 501 and is configured to be executed by the processor 502 to implement the data processing method applied to the convolutional neural network provided by any one of the embodiments of the present application. The related description can be understood by referring to the related description and effects of the steps in the drawings, which will not be described in detail here.

[0119] In the embodiment, the memory 501 and the processor 502 are connected through a bus. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 6 In the figure, only one thick line is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0120] Embodiment nine

[0121] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data processing method applied to the convolutional neural network provided by any one of the embodiments of the present application.

[0122] Embodiment ten

[0123] The embodiment of the present application also provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the data processing method applied to the convolutional neural network provided by any one of the embodiments of the present application.

[0124] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiment described above is only schematic, for example, the division of the module is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or modules, which can be electrical, mechanical or other forms.

[0125] The modules explained as separate components can or can not be physically separate, and the components shown as modules can or can not be physical modules, i.e., can be located in one place, or can be distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0126] In addition, each functional module in each embodiment of the present application can be integrated into one processing module, or each module can be physically present alone, or two or more modules can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of hardware plus software function module.

[0127] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to the processor or controller of a general-purpose computer, a special-purpose computer, or other programmable application-specific data processing device for convolutional neural networks, so that the program codes, when executed by the processor or controller, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program codes can be executed entirely on a machine, partially on a machine, partially on a machine as a separate software package, and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of the present application, the machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media can include one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0129] Furthermore, while operations are depicted in the drawings in a particular, chronological order, this should not be understood as requiring or implying that the operations be performed in that order - and that all illustrated operations be performed to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, while several specific implementation details are contained in the above discussion, these should not be construed as limitations on the scope of the present application, but rather as descriptions of particular implementations. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in subcombination or as separate implementations in accordance with the application.

[0130] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application encompass any and all variations or modifications of the application that fall within the scope of the application along with its principles and practical applications. This application is intended to be illustrative, and not restrictive, of the application's true scope. The true scope of the application is set forth in the following claims.

[0131] It will be understood that the application is not limited to the precise structures hereinbefore described and illustrated in the drawings and that various modifications and changes can be made without departing from the scope thereof. The scope of the application is limited only by the claims that follow.

Claims

1. An electronic device, characterized in that, Includes a processor, and memory: memory for storing executable instructions of the processor; The processor is configured as follows: At least one layer in the convolutional neural network has a channel attention structure; the convolutional kernel corresponding to the layer with the channel attention structure in the convolutional neural network has a dimension greater than 1. Receive the input visual feature data of the current layer, and perform convolution calculation on the input visual feature data to obtain the initial output feature data of the current layer; If the current layer has a channel attention structure, the input visual feature data is input to the channel attention structure to obtain a channel correlation matrix; wherein, the channel attention structure performs predetermined processing on the input visual feature data and outputs the channel correlation matrix; the channel attention structure is a branch structure in a convolutional neural network; wherein, the channel correlation matrix represents the degree of similarity of the transformation results of the channels of the current layer under different shared convolution kernels; The result of multiplying the initial output feature data with the channel correlation matrix is ​​used as the final output feature data of the current layer. The channel attention structure is used to perform the following processing: For each shared convolutional kernel, calculate the convolution result between the shared convolutional kernel and the input visual feature data of the current layer channel; For each shared convolutional kernel, the convolution result between the shared convolutional kernel and the input visual feature data of the current layer channel is calculated to obtain the feature map group corresponding to different shared convolutional kernels; The feature maps corresponding to different shared convolutional kernels are expanded in the spatial dimension to obtain intermediate results corresponding to different shared convolutional kernels. The intermediate results corresponding to different shared convolution kernels are subjected to dot product calculation to obtain the dot product calculation result, and the dot product calculation result is folded in the spatial dimension to obtain the channel correlation matrix.

2. The electronic device according to claim 1, characterized in that, The number of shared convolutional kernels is three; the step of performing dot product calculation on the intermediate results corresponding to different shared convolutional kernels to obtain the dot product calculation result includes: Determine the first shared convolutional kernel and the second shared convolutional kernel among the different shared convolutional kernels; Calculate the dot product between the intermediate result corresponding to the first shared convolution kernel and the intermediate result corresponding to the second shared convolution kernel to obtain the first dot product result; Based on the Softmax function, the first dot product result is processed to obtain the correlation coefficient; Calculate the dot product between the correlation coefficient and the intermediate result corresponding to the third shared convolution kernel to obtain the second dot product result; The second dot product result is processed based on the Sigmoid function to obtain the dot product calculation result.

3. The electronic device according to claim 1, characterized in that, Before multiplying the initial output feature data with the channel correlation matrix, the process also includes: Based on the matrix size corresponding to the initial output feature data, the channel correlation matrix is ​​upsampled; the upsampled channel correlation matrix has the same size as the initial output feature data.

4. The electronic device according to claim 1, characterized in that, The stride of the shared convolutional kernel is no greater than the width of the input visual feature data; wherein the width of the input visual feature data is less than or equal to the length of the input visual feature data.

5. The electronic device according to claim 1, characterized in that, The depth of the shared convolutional kernel is no greater than the depth of the input visual feature data.

6. A data processing apparatus for use in convolutional neural networks, characterized in that, The data processing device is implemented through a physical device that integrates or installs relevant computer programs, wherein at least one layer in the convolutional neural network has a channel attention structure; the convolutional kernel corresponding to the layer with the channel attention structure in the convolutional neural network has a dimension greater than 1, and the data processing device includes: The calculation module is used to receive the input visual feature data of the current layer and perform convolution calculation on the input visual feature data to obtain the initial output feature data of the current layer. An input module is configured to input the input visual feature data into the channel attention structure if the current layer has a channel attention structure, in order to obtain a channel correlation matrix; wherein the channel attention structure outputs the channel correlation matrix by performing predetermined processing on the input visual feature data; the channel attention structure is a branch structure in a convolutional neural network; wherein the channel correlation matrix characterizes the degree of similarity of the transformation results of the channels of the current layer under different shared convolutional kernels; The processing module is used to multiply the initial output feature data with the channel correlation matrix and use the result as the final output feature data of the current layer. The processing module is further configured to: calculate the convolution result of the shared convolution kernel with the input visual feature data of the current layer channel for each shared convolution kernel; calculate the convolution result of the shared convolution kernel with the input visual feature data of the current layer channel for each shared convolution kernel to obtain feature map groups corresponding to different shared convolution kernels; and expand the feature map groups corresponding to different shared convolution kernels in the spatial dimension to obtain intermediate results corresponding to different shared convolution kernels.

Images