Distribution method and device, electronic device and storage medium

Abstract

The present disclosure relates to an allocation method and device, an electronic device and a storage medium, wherein the method comprises: screening the neurons according to the importance of neurons in multiple convolutional layers in the image processing model to obtain a first result; counting the scales of the feature graphs corresponding to each neuron according to the position attribute of each neuron in the first result to obtain an allocation relationship; and allocating the neurons passed through to each feature graph according to the allocation relationship. The embodiment of the present disclosure achieves the purpose of allocating neurons to each feature graph based on the importance of neurons, so that the neurons allocated to each feature graph are more accurate.

Description

[0001] This application is a divisional application of the Chinese patent application submitted to the China Patent Office on February 25, 2019, with application number 201910139007.4 and application name "Network module and distribution method and device, electronic device and storage medium". Technical Field

[0002] The present disclosure relates to the field of computer vision technology, and in particular to a distribution method and device, an electronic device, and a storage medium. Background Art

[0003] Computer vision is an important part of artificial intelligence, and image classification is the foundation of computer vision. Therefore, a good classification network can be used as a backbone network to perform tasks such as support, segmentation, and tracking. In recent years, feature aggregation has become a very effective design method for visual recognition networks. In related technologies, the introduction of multi-scale feature maps can be achieved by connecting feature maps of different depths and using different convolution kernels at the same depth of the network to implement feature aggregation methods. However, when using the above two methods to introduce multi-scale feature maps, the generation and utilization of multi-scale feature maps are relatively limited. Summary of the invention

[0004] The present disclosure proposes a distribution method and device, an electronic device and a storage medium.

[0005] According to one aspect of the present disclosure, there is provided an allocation method for allocating neurons passed by each feature map when there are multiple feature maps in an image processing model. The method comprises:

[0006] According to the importance of neurons in multiple convolutional layers in the image processing model, the neurons are screened to obtain a first result;

[0007] wherein the plurality of convolutional layers are at the same depth of the image processing model, and

[0008] Each convolutional layer is used to process the feature maps of different scales;

[0009] The first result includes a plurality of neurons;

[0010] According to the position attribute of each neuron in the first result, the scale of the feature map corresponding to each neuron is counted to obtain a distribution relationship;

[0011] Wherein, the position attribute represents the convolutional layer to which each neuron belongs;

[0012] The allocation relationship represents the corresponding relationship between each of the feature maps and the neurons through which the feature maps pass;

[0013] According to the allocation relationship, each feature map is allocated the neurons it passes through.

[0014] In a possible implementation, the image processing model includes at least one network module, and the network module includes:

[0015] A first network layer, a second network layer, and a third network layer cascaded in sequence;

[0016] The first network layer is used to process the input image to obtain a first feature map;

[0017] The second network layer includes a plurality of parallel branches;

[0018] Each branch includes a first sampling layer;

[0019] The first sampling layer is used to downsample the first feature map to obtain a second feature map;

[0020] The scales of the second feature maps obtained by the first sampling layers in different branches are different;

[0021] The third network layer is used to merge the feature map data output by each of the branches.

[0022] In a possible implementation, before screening the neurons in the plurality of convolutional layers in the image processing model according to their importance, the method further includes:

[0023] Determine the number of branches in each network module, and construct the image processing model according to the preset number of the network modules;

[0024] Training the image processing model to obtain a scale parameter of a batchnorm layer in the image processing model;

[0025] The scale parameter represents the importance of neurons in the convolutional layers of each branch in the network module.

[0026] In a possible implementation, the step of screening the neurons according to the importance of neurons in a plurality of convolutional layers in the image processing model to obtain the first result includes:

[0027] Sorting the neurons of the plurality of convolutional layers according to a scale parameter of a batchnorm layer obtained by pre-training the image processing model to obtain a first sequence;

[0028] Wherein, the first sequence represents the arrangement order of neurons in the plurality of convolutional layers;

[0029] Determining the number of neurons to be used for processing the plurality of feature maps according to a preset calculation amount;

[0030] According to the determined number of neurons to be used, required neurons are extracted in sequence from the first sequence to obtain the first result.

[0031] In a possible implementation, after allocating the neurons passed by each feature map according to the allocation relationship, the method further includes:

[0032] Determining a first network structure of the image processing model according to the allocation relationship;

[0033] The image processing model of the first network structure is trained.

[0034] In a possible implementation manner, the first sampling layer is a pooling layer.

[0035] In a possible implementation, the pooling layer is a maximum pooling layer.

[0036] In a possible implementation, each branch further includes a first convolutional layer and a second sampling layer sequentially cascaded with the first sampling layer;

[0037] The first convolution layer is used to perform a convolution operation on the second feature map to obtain a third feature map;

[0038] The second sampling layer is used to restore the scale of the third feature map to the scale of the first feature map.

[0039] In a possible implementation, the second network layer further includes an original scale branch; the original scale branch includes a second convolutional layer;

[0040] The second convolution layer is used to perform a convolution operation on the first feature map, and input the feature map data obtained by the convolution operation into the third network layer;

[0041] The third network layer is further used to merge the feature map data output by the multiple parallel branches in the second network layer and the feature map data output by each of the original scale branches.

[0042] According to one aspect of the present disclosure, there is also provided an allocation device for allocating neurons passed through each feature map when there are multiple feature maps in the image processing model, the device comprising:

[0043] A screening module, used for screening the neurons in the plurality of convolutional layers in the image processing model according to their importance, to obtain a first result;

[0044] wherein the plurality of convolutional layers are at the same depth of the image processing model, and

[0045] Each convolutional layer is used to process the feature maps of different scales;

[0046] The first result includes a plurality of neurons;

[0047] A statistical module, used for counting the scale of the feature map corresponding to each neuron according to the position attribute of each neuron in the first result, to obtain a distribution relationship;

[0048] Wherein, the position attribute represents the convolutional layer to which each neuron belongs;

[0049] The allocation relationship represents the corresponding relationship between each of the feature maps and the neurons through which the feature maps pass;

[0050] An allocation module is used to allocate neurons passed by each feature map according to the allocation relationship.

[0051] According to one aspect of the present disclosure, there is also provided an electronic device, including:

[0052] processor;

[0053] a memory for storing processor-executable instructions;

[0054] Wherein, the processor is configured to: execute any of the above methods.

[0055] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above-mentioned allocation method is implemented.

[0056] In the embodiment of the present disclosure, the neurons are screened according to the importance of neurons in multiple convolutional layers in the image processing model, and then the scale of the feature map corresponding to each neuron is determined according to the position of each neuron in the first result obtained by the screening (that is, the feature map to be processed by each neuron is determined), so as to obtain the corresponding allocation relationship. Finally, each feature map is allocated to the neurons it passes through according to the determined allocation relationship, thereby achieving the purpose of allocating neurons to each feature map based on the importance of neurons. This allocation method is data-driven, and the allocation relationship determined for different data sets is different. Compared with the method of artificial experience setting in the related art, the allocation method of the embodiment of the present disclosure makes the neurons ultimately allocated to each feature map more accurate.

[0057] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0058] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0060] Figure 1 A schematic diagram showing the structure of a network module according to an embodiment of the present disclosure is shown;

[0061] Figure 2 A schematic diagram showing the structure of a network module according to another embodiment of the present disclosure is shown;

[0062] Figure 3 A schematic diagram showing the structure of a network module according to another embodiment of the present disclosure is shown;

[0063] Figure 4 A flow chart showing an allocation method according to an embodiment of the present disclosure;

[0064] Figure 5 A block diagram showing a distribution device according to an embodiment of the present disclosure;

[0065] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0066] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0067] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0068] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0069] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0070] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0071] Figure 1 A schematic diagram of the structure of a network module according to an embodiment of the present disclosure is shown.

[0072] See also Figure 1 , the network module of the embodiment of the present disclosure includes a first network layer, a second network layer and a third network layer which are cascaded in sequence. Among them, the first network layer is used to process the input image to obtain a first feature map. The second network layer includes a plurality of parallel branches, each of which includes a first sampling layer. The first sampling layer is used to downsample the first feature map to obtain a second feature map. Among them, the scales of the second feature maps obtained by the first sampling layers in different branches are different. The third network layer is used to merge the feature map data output by each branch, so that the network layer located at the next level of the third network layer can continue to perform image processing operations. Here, it should be pointed out that the third network layer can be a concatenation layer.

[0073] Therefore, by setting the above-mentioned network module, multiple parallel branches are built in the second network layer in the network module, and the first sampling layer in each branch downsamples the first feature map output by the first network layer, so that each different first sampling layer constructs a second feature map with a different scale, thereby achieving the purpose of directly constructing multiple feature maps of different scales in the network module.

[0074] Compared to the related art, by connecting feature maps of different depths and using different convolution kernels at the same depth of the network, the network module of the embodiment of the present disclosure can directly construct feature maps of different scales through the first sampling layer in each branch, that is, construct feature maps of various scales through downsampling operations, which makes it possible to construct feature maps of various scales according to actual conditions using the network module of the embodiment of the present disclosure. Therefore, this effectively improves the diversity of feature maps, and can also make the scale variation range of the obtained feature maps larger and more diverse. At the same time, the multi-branch setting also brings a more diverse receptive field, so that the network module can effectively improve the accuracy of classification and detection when applied to classification and detection tasks.

[0075] It should be noted that the number of branches set in the second network layer of the above network module can be specifically set according to the actual situation. For example, the number of branches can be two, three, five, ten, etc. That is to say, in the network module of the embodiment of the present disclosure, the number of branches in the second network layer can be determined according to its specific computing amount. Therefore, the number of branches is not limited here.

[0076] In addition, it should be noted that the scale of the feature map mentioned in the embodiments of the present disclosure can be the physical size of the feature map, or the size of the effective part of the image (for example, although the physical size of the image is the same, the pixel values ​​of some pixels of the image have been processed by means of, but not limited to, zeroing, etc., and the part composed of other pixels except these processed pixels is equivalent to the effective part, and the size of the effective part is smaller than the physical size of the image), etc., but not limited to this.

[0077] In a possible implementation, the first sampling layer may be a pooling layer, or may be other network layers that can perform various operations (such as zooming in, zooming out, etc.) on the image. In other words, the first sampling layer only needs to process the first feature map so that the second feature map obtained after the processing can have different scales. This effectively improves the structural flexibility of the network module of the embodiment of the present disclosure, thereby making it easier to construct the network module.

[0078] Figure 2 A schematic structural diagram of a network module according to another embodiment of the present disclosure is shown.

[0079] See also Figure 2In a possible implementation, when the first sampling layer is a pooling layer, the pooling layer may be a maximum pooling layer (max pool). That is, the first sampling layer is implemented by the maximum pooling layer. Among them, the first network layer may be a convolutional layer, and the convolution kernel of the convolutional layer may be 1*1 in size (i.e., 1*1 convolutional layer). Thus, after the first network layer (e.g., 1*1 convolutional layer) processes the input image, a corresponding first feature map is obtained. At this time, the first feature map has a first scale. Then, the first sampling layer (e.g., maximum pooling layer) of each branch in the second network layer downsamples the first feature map with the first scale (i.e., the maximum pooling layer performs maximum pooling processing on the first feature map), and obtains a second feature map with different scales, thereby achieving the purpose of directly constructing feature maps with different scales in the network module.

[0080] By using the maximum pooling layer as the first sampling layer to downsample the first feature map, since the maximum pooling can effectively reduce the size of the feature map, the subsequent image processing operations (such as convolution operations) in the corresponding small-scale branches will consume less computational effort, which effectively reduces the amount of computation in each branch and reduces power consumption.

[0081] In a possible implementation, each branch also includes a first convolutional layer and a second sampling layer which are sequentially cascaded with the first sampling layer. The first convolutional layer is used to perform a convolution operation on the second feature map to obtain a third feature map. It should be noted here that the first convolutional layer can be a convolutional layer with convolution kernels of different sizes (such as: 3*3 convolutional layer, 3*3conv). In addition, the convolution kernel sizes of the first convolutional layer in each branch can be the same or different. The second sampling layer is used to restore the scale of the third feature map to the scale of the first feature map. It should be noted that the second sampling layer can be an upsample layer.

[0082] That is to say, after downsampling the first feature map in each first sampling layer to obtain the second feature map with different scales, the first convolution layer performs a convolution operation on the second feature map with different scales to realize convolution processing on the feature map.

[0083] At the same time, due to the downsampling operation of the first sampling layer on the first feature map, the scale of the obtained second feature map changes compared with the first feature map. In each branch, the scale of the third feature map obtained by convolving the second feature map by the first convolution layer may also change with the scale of the second feature map. Therefore, in order to smoothly perform other image processing subsequently, the second sampling layer is required to upsample the third feature map so that the scale of the third feature map is restored to the original scale (i.e., the scale of the first feature map).

[0084] Therefore, the network module of the embodiment of the present disclosure realizes the construction of feature maps of different scales through upsampling and downsampling operations, so that multi-scale features can be extracted effectively and efficiently.

[0085] Figure 3 A schematic structural diagram of a network module according to another embodiment of the present disclosure is shown.

[0086] See also Figure 3 , the network module of the embodiment of the present disclosure, the second network layer may also include an original proportion branch. Among them, the original proportion branch is in parallel with the multiple parallel branches mentioned above, and the original proportion branch does not change the scale of the first feature map. Among them, the original proportion branch includes a second convolutional layer, and the second convolutional layer is used to perform a convolution operation on the first feature map, and input the feature map data obtained by the convolution operation into the third network layer, and the feature data obtained by the convolution has the same scale as the first feature map. Here, it should be noted that the second convolutional layer can be a 3*3 convolutional layer (i.e., 3*3conv). Correspondingly, the third network layer is also used to merge the feature map data output by multiple parallel branches in the second network layer and the feature map data output by the original proportion branch.

[0087] That is, by setting the original scale branch in the second network layer, the second convolutional layer in the original scale branch directly performs a convolution operation on the first feature map. On the basis of effectively increasing the scale of the feature map, the processing of the first feature map of the original scale is further guaranteed, which ensures the integrity and accuracy of the processed image data and avoids the loss of some features in the first feature map.

[0088] In addition, it should be pointed out that the network module of the embodiment of the present disclosure can be used as the smallest basic unit (can be referred to as: block) in the neural network structure. That is, a network structure with different depths can be constructed by repeatedly stacking any of the above network modules (blocks). Among them, the constructed network structure can be a convolutional neural network.

[0089] According to another aspect of the present disclosure, an allocation method is also provided. The allocation method of the present disclosure is used to allocate neurons passed by each feature map when there are multiple feature maps in the image processing model. Each feature map has a different scale. The image processing model can be a convolutional neural network model.

[0090] Figure 4 A flow chart showing a method for allocating data according to an embodiment of the present disclosure is shown. Figure 4 , the allocation method disclosed herein comprises:

[0091] Step S100, according to the importance of neurons in multiple convolutional layers in the image processing model, the neurons are screened to obtain a first result. It should be noted that the multiple convolutional layers here are at the same depth of the image processing model (that is, the multiple convolutional layers are located at the same layer of the image processing model), and each convolutional layer is used to process feature maps of different scales. At the same time, the first result contains multiple neurons.

[0092] Step S200, according to the position attribute of each neuron in the first result, the scale of the feature map corresponding to each neuron is counted to obtain the allocation relationship. Among them, the position attribute represents the convolution layer to which each neuron belongs. In other words, the position attribute is used to determine which convolution layer the neuron belongs to. The allocation relationship represents the corresponding relationship between each feature map and the neurons through which the feature map passes. That is, the allocation relationship can be used to determine which neurons each feature map should pass through for processing and calculation.

[0093] Step S300, assigning neurons to each feature map according to the assignment relationship.

[0094] Therefore, the above-disclosed allocation method screens neurons according to the importance of neurons in multiple convolutional layers in the image processing model, and then determines the scale of the feature map corresponding to each neuron according to the position of each neuron in the first result obtained by screening (that is, determines the feature map to be processed by each neuron), thereby obtaining the corresponding allocation relationship. Finally, each feature map and the neurons it passes through are allocated according to the determined allocation relationship, thereby achieving the purpose of allocating neurons to each feature map based on the importance of neurons. This allocation method is data-driven, and the allocation relationship determined for different data sets is different. Compared with the method of artificial experience setting in the related art, the allocation method of the embodiment of the present disclosure makes the neurons ultimately allocated to each feature map more accurate.

[0095] It should be noted that the allocation method provided in the embodiment of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, it can also be executed by a processor, such as the processor executing any allocation method mentioned in the embodiment of the present disclosure by calling the corresponding instructions stored in the memory. It will not be repeated below. The processor can be a general-purpose processor or an artificial intelligence processor.

[0096] In addition, it should be pointed out that in the allocation method of the embodiment of the present disclosure, there should be multiple feature maps with different scales in the image processing model. That is to say, the image processing model can be a first type of network structure. The first type of network structure is: a network structure that introduces multi-scale feature maps by combining shallow feature maps and deep feature maps by adding residuals between feature maps of different depths. The image processing model can also be a second type of network structure. The second type of network structure is a network structure that introduces multi-scale feature maps by using different convolution kernels at the same depth. The image processing model can also be a third type of network structure. The third type of network structure is: a network structure including any of the network modules (blocks) described above (that is, a network structure with a certain depth constructed by repeatedly stacking network modules).

[0097] In a possible implementation, the image processing model may include any of the above-mentioned network modules, wherein the number of the network modules is at least one.

[0098] Therefore, before screening neurons according to the importance of neurons in multiple convolutional layers in the image processing model, the following may also be included:

[0099] The number of branches in each network module is determined, and the network module is constructed according to the determined number of branches in the network module. Then, the image processing model is constructed according to the number of pre-set network modules. The number of branches in each network model can be determined according to the amount of calculation required in the actual situation.

[0100] The image processing model is trained to obtain the scale parameter of the batchnorm layer in the image processing model, wherein the scale parameter represents the importance of the neurons in the convolutional layer of each branch in the network module.

[0101] That is, the allocation method of the embodiment of the present disclosure can be applied to an image processing model including any of the network modules described above. When the image processing model includes a network module, the multiple convolutional layers can be the first convolutional layer in each branch of the second network layer in the network module and the second convolutional layer in the original scale branch.

[0102] That is to say, when the image processing model includes any of the network modules described above, the multiple convolutional layers at the same depth in the image processing model can be the first convolutional layer in each branch of the second network layer in the network module, and / or the second convolutional layer in the original scale branch of the second network layer.

[0103] Here, it should also be pointed out that, since the image processing model includes any of the aforementioned network modules, the number of network modules is at least one. When the number of network modules is multiple, the allocation of neurons is based on the allocation of neurons in multiple convolutional layers (such as the first convolutional layer and the second convolutional layer) in each network module (block). It should also be pointed out that when the number of network modules is multiple, multiple network modules can be stacked in sequence to construct an image processing model. That is, multiple network modules can be set in series, and corresponding network layers can be set between each two adjacent network modules according to actual needs. No specific limitation is made here.

[0104] For example, when the image processing model includes 20 network modules (i.e., a network structure with a certain depth constructed by stacking 20 blocks), the neurons of the first convolutional layer in each branch of the second network layer in the 20 blocks and the second convolutional layer in the original proportion branch can be screened and allocated respectively. The neuron allocation process of multiple convolutional layers in the 20 blocks can be performed simultaneously or sequentially. This is not limited here.

[0105] In a possible implementation, screening neurons according to the importance of neurons in multiple convolutional layers in the image processing model to obtain the first result may include:

[0106] According to the scale parameter of the batchnorm layer obtained by pre-training the image processing model, the neurons of the multiple convolutional layers are sorted to obtain a first sequence, wherein the first sequence represents the arrangement order of the neurons in the multiple convolutional layers.

[0107] According to the preset calculation amount, the number of neurons to be used for processing multiple feature maps is determined.

[0108] According to the determined number of neurons to be used, the required neurons are extracted in sequence from the first sequence to obtain a first result.

[0109] That is, the scale parameter of the batchnorm layer learned by pre-training the image processing model is used as the criterion, and the neurons in multiple convolutional layers (here, all neurons in multiple convolutional layers) are sorted (wherein the arrangement order can be arranged from high to low) to obtain the corresponding first sequence. At the same time, the number of neurons (the number of neurons to be used) required to process multiple feature maps is determined according to the preset calculation amount (that is, the actual amount of calculation required), and according to the determined number of neurons to be used, the required neurons are extracted in sequence from the first sequence according to the arrangement order of the neurons. Among them, the number of required neurons extracted is consistent with the number of neurons to be used.

[0110] Therefore, by selecting neurons of multiple convolutional layers according to the learned scale parameters, multiple neurons are competitively and adaptively allocated, which effectively improves the accuracy of neuron allocation and also effectively improves the rationality of neuron allocation.

[0111] In addition, it should be noted that in any of the above-mentioned allocation method embodiments, when allocating the neurons passed through each feature map according to the allocation relationship, it may include: retaining the required neurons and deleting the unnecessary neurons.

[0112] As a possible implementation method, after allocating the neurons passed by each feature map according to the allocation relationship, it can also include:

[0113] According to the allocation relationship, the first network structure of the image processing model is determined, and the image processing model of the first network structure is trained to achieve the purpose of optimizing the image processing model, so that the final image processing model can have higher accuracy in classification and detection tasks.

[0114] In order to more clearly illustrate the process of the allocation method of the embodiment of the present disclosure, the following image processing model includes a network module, and the network module is Figure 3 The network structure shown in FIG. 1 is taken as an example for a clearer and more detailed description.

[0115] See also Figure 3 , Figure 3 The second network layer of the network module shown includes an original ratio branch and two branches (the first branch and the second branch). Among them, the original ratio branch, the first branch and the second branch are all set in parallel. The original ratio branch contains a 3*3 convolution layer (i.e., the second convolution layer, 3*3conv), and the first branch and the second branch respectively include a first sampling layer (maximum pooling layer, max pool), a first convolution layer (3*3conv) and a second sampling layer (upsample) cascaded in sequence. When allocating neurons to the image processing model, the neurons of the two first convolution layers and one second convolution layer in the network module it contains are mainly allocated.

[0116] Among them, it can be considered that: the number of neurons in the second convolutional layer in the original proportion branch is 10 (neuron 1, neuron 2, neuron 3...neuron 10 respectively), the number of neurons in the first convolutional layer in the first branch is also 10 (neuron 11, neuron 12, neuron 13...neuron 20 respectively), and the number of neurons in the first convolutional layer in the second branch can also be 10 (neuron 21, neuron 22, neuron 23...neuron 30 respectively).

[0117] At the same time, since the number of branches in the second network layer is three (the original scale branch, the first branch, and the second branch), the number of feature maps constructed is also three. The scale of the feature map in the original scale branch is the original scale, the scale of the feature map constructed in the first branch is the first scale, and the scale of the feature map constructed in the second branch is the second scale.

[0118] After learning the scale parameter of the batchnorm layer by training the image processing model, the 30 neurons are sorted according to the scale parameter to obtain a neuron sequence (i.e., the first sequence). The first sequence obtained is: neuron 1, neuron 2, neuron 3, neuron 4, ..., neuron 28, neuron 29, neuron 30.

[0119] According to the preset calculation amount, it is determined that the number of neurons (i.e., the number of neurons to be used) required by the image processing model in this embodiment to process the feature maps of the three different scales is 15. Therefore, at this time, according to the determined number of neurons to be used, the required neurons (respectively: neuron 1, neuron 2, neuron 3, neuron 4, ..., neuron 14, neuron 15) can be extracted in sequence from the first sequence according to the arrangement order of the neurons, thereby obtaining the first result.

[0120] According to the position attribute of each neuron in the first result, the scale of the feature map corresponding to each neuron is counted to obtain the distribution relationship. That is, according to the respective positions of neuron 1, neuron 2, neuron 3, neuron 4, ..., neuron 14, and neuron 15, the scale of the feature map corresponding to each neuron is determined. In other words, the branch to which each neuron belongs is determined according to its position attribute. Among them, it can be determined that neurons 1 to 10 belong to the original proportion branch, and the scale of the feature map corresponding to these 10 neurons is the original scale, and neurons 10 to 15 belong to the first branch, and the scale of the feature map corresponding to these 5 neurons is the first scale. Therefore, the corresponding distribution relationship (that is, the scale of the feature map corresponding to each neuron 1 to 15) can be obtained.

[0121] After the allocation relationship is determined, the neurons through which each feature map passes can be allocated according to the allocation relationship. That is, neurons 1 to 15 are retained, and the second branch including neurons 20 to 30 is deleted. In other words, the second branch for constructing the feature map of the second scale is discarded. In this way, the allocation of neurons in the network module in this embodiment can be completed.

[0122] In summary, the allocation method of the embodiment of the present invention allocates neurons to each feature map with different scales according to the importance of neurons in multiple convolutional layers at the same depth in the image processing model, so that multiple neurons can be competitively and adaptively allocated, thereby effectively improving the accuracy and rationality of the allocation result. At the same time, it also optimizes the network structure of the image processing model, so that when the optimized image processing model is retrained and applied to classification and detection tasks, the classification and detection accuracy can be effectively improved.

[0123] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not go into details.

[0124] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0125] In addition, the present disclosure also provides a distribution device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any distribution method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0126] Figure 5 A block diagram of a distribution device 100 according to an embodiment of the present disclosure is shown. Figure 5 As shown, the allocation device 100 is used to allocate neurons passed by each feature map when there are multiple feature maps in the image processing model, and the image processing model includes at least one network module described in any of the above; the device 100 includes:

[0127] A screening module 110, configured to screen the neurons in the plurality of convolutional layers in the image processing model according to their importance, to obtain a first result;

[0128] wherein the plurality of convolutional layers are at the same depth of the image processing model, and

[0129] Each convolutional layer is used to process the feature maps of different scales;

[0130] The first result includes a plurality of neurons;

[0131] A statistical module 120, configured to count the scale of the feature graph corresponding to each neuron according to the position attribute of each neuron in the first result, so as to obtain a distribution relationship;

[0132] Wherein, the position attribute represents the convolutional layer to which each neuron belongs;

[0133] The allocation relationship represents the corresponding relationship between each of the feature maps and the neurons through which the feature maps pass;

[0134] The allocation module 130 is used to allocate the neurons passed by each feature map according to the allocation relationship.

[0135] In a possible implementation, the method further includes:

[0136] A model building module, used for determining the number of branches in each network module before the screening module screens the neurons according to the importance of the neurons in the multiple convolutional layers in the image processing model, and building the image processing model according to the preset number of the network modules;

[0137] A first training module is used to train the image processing model to obtain a scale parameter of a batchnorm layer in the image processing model;

[0138] The scale parameter represents the importance of neurons in the convolutional layers of each branch in the network module.

[0139] In a possible implementation, the screening module 110 includes:

[0140] A sorting submodule, used for sorting the neurons of the plurality of convolutional layers according to the scale parameter of the batchnorm layer obtained by pre-training the image processing model, to obtain a first sequence;

[0141] Wherein, the first sequence represents the arrangement order of neurons in the plurality of convolutional layers;

[0142] A neuron number determination submodule, used to determine the number of neurons to be used for processing the plurality of feature maps according to a preset calculation amount;

[0143] The neuron extraction submodule is used to extract required neurons from the first sequence in sequence according to the determined number of neurons to be used, so as to obtain the first result.

[0144] In a possible implementation, the method further includes:

[0145] A network structure determination module, configured to determine a first network structure of the image processing model according to the allocation relationship after the allocation module allocates the neurons passed through each feature map according to the allocation relationship;

[0146] The second training module is used to train the image processing model of the first network structure.

[0147] In some embodiments, the functions or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiment. The specific implementation thereof may refer to the description of the above method embodiment. For the sake of brevity, it will not be repeated here.

[0148] The embodiment of the present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, any of the above allocation methods is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0149] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute any of the allocation methods described above.

[0150] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0151] Figure 6 8 is a block diagram of an electronic device 800 according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0152] Reference Figure 6 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0153] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0154] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0155] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0156] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0157] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0158] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0159] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0160] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0161] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0162] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0163] Figure 7 1 is a block diagram of an electronic device 1900 according to an exemplary embodiment. For example, the electronic device 1900 may be provided as a server. Figure 7, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0164] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0165] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0166] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0167] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0168] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0169] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0170] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0171] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0172] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0173] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0174] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A distribution method, characterized in that: When there are multiple feature maps in an image processing model, assigning neurons to each feature map, the method comprising: According to the importance of neurons in multiple convolutional layers in the image processing model, the neurons are screened to obtain a first result; wherein the plurality of convolutional layers are at the same depth of the image processing model, and Each convolutional layer is used to process the feature maps of different scales; The first result includes a plurality of neurons; According to the position attribute of each neuron in the first result, the scale of the feature map corresponding to each neuron is counted to obtain a distribution relationship; Wherein, the position attribute represents the convolutional layer to which each neuron belongs; The allocation relationship represents the corresponding relationship between each of the feature maps and the neurons through which the feature maps pass; According to the allocation relationship, each feature map is allocated the neurons it passes through.

2. The method according to claim 1, characterized in that The image processing model includes at least one network module, and the network module includes: A first network layer, a second network layer, and a third network layer cascaded in sequence; The first network layer is used to process the input image to obtain a first feature map; The second network layer includes a plurality of parallel branches; Each branch includes a first sampling layer; The first sampling layer is used to downsample the first feature map to obtain a second feature map; The scales of the second feature maps obtained by the first sampling layers in different branches are different; The third network layer is used to merge the feature map data output by each of the branches.

3. The method according to claim 2, characterized in that Before screening the neurons according to the importance of the neurons in the plurality of convolutional layers in the image processing model, the method further comprises: Determine the number of branches in each network module, and construct the image processing model according to the preset number of the network modules; Training the image processing model to obtain a scale parameter of a batchnorm layer in the image processing model; The scale parameter represents the importance of neurons in the convolutional layers of each branch in the network module.

4. The method according to any one of claims 1 to 3, characterized in that: The step of screening the neurons according to the importance of the neurons in the plurality of convolutional layers in the image processing model to obtain the first result includes: Sorting the neurons of the plurality of convolutional layers according to a scale parameter of a batchnorm layer obtained by pre-training the image processing model to obtain a first sequence; Wherein, the first sequence represents the arrangement order of neurons in the plurality of convolutional layers; Determining the number of neurons to be used for processing the plurality of feature maps according to a preset calculation amount; According to the determined number of neurons to be used, required neurons are extracted in sequence from the first sequence to obtain the first result.

5. The method according to claim 4, characterized in that After allocating the neurons passed by each feature map according to the allocation relationship, the method further includes: Determining a first network structure of the image processing model according to the allocation relationship; The image processing model of the first network structure is trained.

6. The method according to claim 2, characterized in that Each branch also includes a first convolutional layer and a second sampling layer sequentially cascaded with the first sampling layer; The first convolution layer is used to perform a convolution operation on the second feature map to obtain a third feature map; The second sampling layer is used to restore the scale of the third feature map to the scale of the first feature map.

7. The method according to any one of claims 2 and 6, characterized in that: The second network layer further includes an original scale branch; the original scale branch includes a second convolutional layer; The second convolution layer is used to perform a convolution operation on the first feature map, and input the feature map data obtained by the convolution operation into the third network layer; The third network layer is further used to merge the feature map data output by the multiple parallel branches in the second network layer and the feature map data output by each of the original scale branches.

8. A dispensing device, characterized in that: When there are multiple feature maps in an image processing model, the device allocates neurons passed by each feature map, and includes: A screening module, used for screening the neurons in the plurality of convolutional layers in the image processing model according to their importance, to obtain a first result; wherein the plurality of convolutional layers are at the same depth of the image processing model, and Each convolutional layer is used to process the feature maps of different scales; The first result includes a plurality of neurons; A statistical module, used for counting the scale of the feature map corresponding to each neuron according to the position attribute of each neuron in the first result, to obtain a distribution relationship; Wherein, the position attribute represents the convolutional layer to which each neuron belongs; The allocation relationship represents the corresponding relationship between each of the feature maps and the neurons through which the feature maps pass; An allocation module is used to allocate neurons passed by each feature map according to the allocation relationship.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to: execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images