Visual task processing method and model training method, device and electronic system thereof

By using the first subnet and the second subnet to generate and select filters in the convolution network, feature extraction is performed for each position point, and large amount of computing and redundant feature extraction caused by the increase in the number of filters in the prior art are solved. The problem improves feature extraction efficiency and network model accuracy.

CN111860764BActive Publication Date: 2024-12-03MEGVII (BEIJING) TECHNOLOGY CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202010509859.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-06-05
Publication Date
2024-12-03
Estimated Expiration
2040-06-05

AI Technical Summary

Technical Problem

When extracting features by convolutional networks in existing deep learning networks, although increasing the number of filters can extract richer features, it will lead to large amount of computing and redundant feature extraction, reducing the efficiency and accuracy of the network model.

Method used

By introducing a first subnet and a second subnet in the convolutional network, it is used to generate and select filters respectively, and determine a designated filter based on data for each position point, and perform feature extraction, avoiding the simultaneous increase in the number of filters Improved feature extraction efficiency.

Benefits of technology

Without increasing the number of filters, different filters are used to extract features, which improves the efficiency of feature extraction and the accuracy of network model, simplifies the network structure, and reduces the computational complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111860764B_ABST
    Figure CN111860764B_ABST
Patent Text Reader

Abstract

The present invention provides a visual task processing method and a model training method, device and electronic system thereof, which extracts feature data of an image to be processed through a preset network model; wherein the network model includes at least one layer of convolutional network; a data matrix including multiple position points is input into the convolutional network; for each position point, a designated filter of the position point is determined according to the data at the position point; and the feature data of the position point is extracted through the designated filter. This method uses different filters to extract features for different data without increasing the number of filters, so that the extracted features are more effective, thereby improving the efficiency of feature extraction, which is conducive to improving the accuracy and other performance of the network model as a whole.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a visual task processing method and a model training method, device and electronic system thereof. Background Art

[0002] In deep learning network models, convolutional networks are usually used to extract feature data; a layer of convolutional networks usually contains multiple filters; each filter extracts the features of each local position of the input data of the convolutional network in the spatial dimension in the form of a sliding window, thereby obtaining the feature data of a channel; therefore, the number of channels for outputting the feature data of the convolutional network is the same as the number of filters of the convolutional network. Each filter in a layer of convolutional networks is used to extract specific features. In order to extract richer features, the usual approach is to increase the number of filters in the convolutional network. However, on the one hand, this will lead to a large amount of computation for the network model, and on the other hand, many redundant features will be extracted, resulting in a low efficiency in extracting features for the network model, which is not conducive to improving the accuracy and other performance of the network model as a whole. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a visual task processing method and a model training method, device and electronic system thereof to improve the efficiency of feature extraction by a network model.

[0004] In a first aspect, an embodiment of the present invention provides a visual task processing method, the method comprising: obtaining an image to be processed, inputting the image to be processed into a preset network model, and outputting feature data corresponding to the image to be processed; based on the feature data, completing a preset visual task; wherein the network model comprises at least one layer of convolutional network; the data matrix input to the convolutional network comprises multiple position points; for each position point, extracting the feature data of the position point through a specified filter of the position point; the specified filter is determined based on the data at the position point.

[0005] Furthermore, the convolutional network includes a first sub-network and a second sub-network; the first sub-network is used to: generate multiple filters according to the data matrix; the second sub-network is used to: for each position point in the data matrix, select a specified filter from multiple filters according to the data at the position point, so as to extract the feature data of the position point through the specified filter.

[0006] Furthermore, the first subnetwork is specifically used to: perform an average pooling operation on the data matrix to obtain a pooling result; wherein the scale of the pooling result matches the scale of the filter to be generated; perform a first convolution operation on the pooling result to obtain a first convolution result; wherein the number of channels of the first convolution result is associated with the number of types of filters to be generated; perform a second convolution operation on the first convolution result to obtain a second convolution result; wherein the second convolution result includes multiple filters; the number of channels of each filter is associated with the number of channels of the data matrix and the number of channels of the convolution network output data.

[0007] Furthermore, the scale of the pooling result is the same as the scale of the filter to be generated; the number of channels of the first convolution result is equal to the square of the number of types of filters to be generated; the number of channels of each filter is equal to the product of the number of channels of the data matrix and the number of channels of the convolutional network output data.

[0008] Furthermore, the second subnetwork is specifically used to: perform a third convolution operation on the data matrix to obtain a third convolution result; wherein the number of channels of the third convolution result is associated with the number of types of the multiple filters; for each position point in the third convolution result, perform multi-classification processing on the data at the position point to obtain a classification result; the classification result includes a probability value of using each filter to extract feature data at the position point; and determine the filter with the largest probability value as the target filter for the position point.

[0009] Furthermore, the number of channels of the third convolution result is the same as the number of types of the multiple filters.

[0010] In the second aspect, an embodiment of the present invention provides a network model training method for visual task processing, the method comprising: inputting a sample image into an initial model; the initial model comprises at least one layer of convolutional network; for the convolutional network, generating a plurality of filters according to a data matrix input into the convolutional network; for each position point of the data matrix, determining a designated filter for the position point from a plurality of filters according to the data at the position point; extracting feature data of the position point through the designated filter; calculating a loss value based on a preset loss function and feature data output by the initial model; iteratively training the initial model according to the loss value to obtain a trained network model.

[0011] Furthermore, the step of determining a designated filter for the position point from a plurality of filters according to the data at the position point comprises: performing multi-classification processing on the data at the position point to obtain a classification result; wherein the classification result comprises a probability value of extracting feature data at the position point using each filter; determining the filter with the largest probability value as the target filter for the position point; and iteratively training the initial model according to the loss value to obtain a trained network model comprises: for the convolutional network, determining, according to the classification result, the target filter for each position point when the convolutional network performs back propagation; and adjusting the network parameters of the convolutional network according to the loss value and the target filter for each position point to obtain a trained network model.

[0012] Furthermore, based on the classification results, the step of determining the target filter for each position point when the convolutional network performs back propagation includes: performing the following operations for each position point: for each filter, multiplying the filter by the probability value of extracting feature data at the position point using the filter to obtain a product result; summing the product results of each filter to obtain the target filter for the position point.

[0013] Furthermore, the step of adjusting the network parameters of the convolutional network according to the loss value and the target filter of each position point includes: calculating the feature data of each position point according to the target filter of each position point; calculating the gradient of the classification result according to the feature data of each position point; and adjusting the network parameters of the convolutional network according to the gradient of the classification result and the loss value.

[0014] In a third aspect, an embodiment of the present invention provides a visual task processing device, which includes: a data output module, used to obtain an image to be processed, input the image to be processed into a preset network model, and output feature data corresponding to the image to be processed; a task processing module, used to complete a preset visual task based on the feature data; wherein the network model includes at least one layer of convolutional network; the data matrix input to the convolutional network includes multiple position points; for each position point, the feature data of the position point is extracted through a specified filter of the position point; the specified filter is determined based on the data at the position point.

[0015] In a fourth aspect, an embodiment of the present invention provides a network model training device for visual task processing, the device comprising: an input module for inputting a sample image into an initial model; the initial model comprising at least one layer of convolutional network; a generation module for generating a plurality of filters for the convolutional network according to a data matrix input into the convolutional network; for each position point in the data matrix, determining a designated filter for the position point from a plurality of filters according to the data at the position point; extracting feature data of the position point through the designated filter; a calculation module for calculating a loss value based on a preset loss function and feature data output by the initial model; and a training module for iteratively training the initial model according to the loss value to obtain a trained network model.

[0016] In a fifth aspect, an embodiment of the present invention provides an electronic system, which includes: a processing device and a storage device; a computer program is stored on the storage device, and when the computer program is run by the processed device, it executes the visual task processing method of any embodiment of the first aspect and the network model training method applied to visual task processing of any embodiment of the second aspect.

[0017] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processing device, the steps of the visual task processing method of any embodiment of the first aspect and the network model training method for visual task processing of any embodiment of the second aspect are executed.

[0018] The embodiments of the present invention bring the following beneficial effects:

[0019] The embodiment of the present invention provides a visual task processing method and its model training method, device and electronic system, which extracts feature data of an image to be processed through a preset network model; wherein the network model includes at least one layer of convolutional network; a data matrix including multiple position points is input into the convolutional network; for each position point, a designated filter of the position point is determined according to the data at the position point; and the feature data of the position point is extracted through the designated filter. In this method, for each position point in the data matrix input into the convolutional network, a designated filter of the position point is determined according to the data at the position point, and then the feature data at the position point is extracted based on the designated filter. This method uses different filters to extract features for different data without increasing the number of filters, so that the extracted features are more effective, thereby improving the efficiency of feature extraction, which is conducive to improving the accuracy and other performance of the network model as a whole.

[0020] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0021] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A schematic diagram of the structure of an electronic system provided by an embodiment of the present invention;

[0024] Figure 2 A flowchart of a visual task processing method provided by an embodiment of the present invention;

[0025] Figure 3 A flowchart of an application method of a first sub-network in a visual task processing method provided by an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of a filter generation method provided by an embodiment of the present invention;

[0027] Figure 5 A flowchart of an application method of a second sub-network in a visual task processing method provided by an embodiment of the present invention;

[0028] Figure 6 A schematic diagram of another filter generation method provided by an embodiment of the present invention;

[0029] Figure 7 A schematic diagram of a multi-filter convolution provided by an embodiment of the present invention;

[0030] Figure 8 A flowchart of a network model training method for visual task processing provided by an embodiment of the present invention;

[0031] Fig. 9 A flowchart of another network model training method for visual task processing provided by an embodiment of the present invention;

[0032] Fig.10A schematic diagram of the structure of a visual task processing device provided by an embodiment of the present invention;

[0033] Fig.11 A schematic diagram of the structure of a network model training device for visual task processing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0035] In the prior art, feature extraction usually shares filters in the entire spatial dimension, ignoring the relationship between features; in order to improve network performance, the number of filters is usually increased by increasing the number of channels, but this will increase the amount of network computation, making the network run for a long time, which is not conducive to practical application; in addition, multiple filters repeatedly extract features in the entire spatial dimension, which will extract a large number of redundant features, resulting in low efficiency in feature extraction by the network model, which is not conducive to improving the accuracy and other performance of the network model as a whole; secondly, if a filter is designed separately for each position, the structure will be too complicated for similar information in the spatial dimension.

[0036] Based on this, an embodiment of the present invention provides a visual task processing method and a model training method, device and electronic system thereof. This technology can be applied to a variety of devices such as security equipment, computers, mobile phones, cameras, tablet computers, vehicle terminal equipment, etc. This technology can be implemented using relevant software and hardware, and is described below through an embodiment.

[0037] Embodiment 1:

[0038] First, refer to Figure 1 An example electronic system 100 is described to implement the visual task processing method and model training method, device, and electronic system of the embodiment of the present invention.

[0039] like Figure 1 The electronic system 100 includes one or more processing devices 102, one or more storage devices 104, an input device 106, an output device 108, and may also include one or more image acquisition devices 110. These components are interconnected through a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1The components and structure of the electronic system 100 shown are merely exemplary and not limiting. The electronic system may also have other components and structures as required.

[0040] The processing device 102 can be a gateway, or a smart terminal, or a device including a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It can process data of other components in the electronic system 100 and control other components in the electronic system 100 to perform desired functions.

[0041] The storage device 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processing device 102 may run the program instructions to implement the client functions and / or other desired functions in the embodiments of the present invention (implemented by the processing device) described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0042] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0043] The output device 108 may output various information (eg, images, sounds, or data) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

[0044] The image acquisition device 110 may acquire preview video frames or image data (such as images to be processed or training images), and store the acquired preview video frames or image data in the storage device 104 for use by other components.

[0045] Exemplarily, the components in the example electronic system for implementing the visual task processing method and its model training method, device and electronic system according to the embodiment of the present invention can be integrated or dispersed, such as integrating the processing device 102, the storage device 104, the input device 106 and the output device 108 into one, and setting the image acquisition device 110 at a specified position where the image can be acquired. When the components in the above electronic system are integrated, the electronic system can be implemented as an intelligent terminal such as a camera, a smart phone, a tablet computer, a computer, a vehicle-mounted terminal, a camera, etc.

[0046] Embodiment 2:

[0047] This embodiment provides a visual task processing method, such as Figure 2 As shown, the method comprises the following steps:

[0048] Step S202, obtaining an image to be processed, inputting the image to be processed into a preset network model, and outputting feature data corresponding to the image to be processed;

[0049] The above-mentioned image to be processed may be an image containing faces, objects, landscapes, and texts, such as images of faces and pedestrians captured by a camera, or images of objects placed in shopping malls and supermarkets, or images of various buildings, natural landscapes, or various text images, etc.; in addition, the above-mentioned image to be processed may be a feature map containing multiple channels, for example, according to the basic color of the image, the image can be divided into feature maps of three colors: red, green, and blue, i.e., a feature map of three channels. The above-mentioned preset network model may be a network model such as CNN (Convolutional Neural Networks), RNN (Recurrent Neural Network), DNN (Deep Neural Network), etc.; the network model may generally include a multi-layer convolutional network, and may also include multiple filters. In actual implementation, the feature data of multiple position points in the image to be processed are extracted through the filters in the above-mentioned preset network model.

[0050] Step S204, completing a preset visual task based on the feature data;

[0051] The above-mentioned preset visual tasks may include face recognition, object retrieval, target detection, image semantic segmentation, scene text recognition, human key point detection, video classification, metric learning, etc.; the above-mentioned feature data may be feature data about various types of images such as faces, objects, semantics, scenes, and texts; in reality, the target task of the image to be processed can be identified according to the output feature data; for example, according to the output face feature data, the face contained in the image to be processed can be identified; for another example, according to the output object feature data, the target object can be retrieved from the image to be processed containing multiple types of objects; for another example, according to the output text feature data, semantics or text scenes can be identified from the image to be processed containing multiple texts, etc.; thereby completing the preset visual task.

[0052] Among them, the network model includes at least one layer of convolutional network; the data matrix input to the convolutional network includes multiple position points; for each position point, the feature data of the position point is extracted through the specified filter of the position point; the specified filter is determined according to the data at the position point.

[0053] The above convolutional network can also be called a convolutional layer. Each convolutional layer in the convolutional neural network is composed of several convolutional units. The parameters of each convolutional unit are optimized through the back propagation algorithm. The function of the convolutional layer is convolution operation. The purpose of convolution operation is to extract different features of the input. The first convolutional layer can usually only extract some low-level features such as edges, lines and corners. More layers of the network can iteratively extract more complex features from low-level features. The above data matrix usually refers to image data; the rows of the data matrix correspond to the height of the image (in pixels), the columns of the data matrix correspond to the width of the image (in pixels), and the elements of the data matrix, that is, the above position points, can correspond to the pixels of the image and other data; for example, the elements of the data matrix can correspond to the grayscale values ​​of the pixels in the image.

[0054] In actual implementation, through the convolutional network, multiple filters can be generated according to the characteristics of the data matrix such as shape, scale, data features, and the actual needs of the visual task; among them, different filters can extract different features; in addition, each position point in the data matrix input to the convolutional network can be classified according to the data of the position point through various functions and convolution kernels contained in the convolutional network, such as classification functions, loss functions, etc. According to the data of each position point and the function of different filters in the network to extract different features, the data of each position point can be obtained by using different filters to extract the data of the position point. For example, the accuracy, usage probability, operation speed, extraction effect, etc. of each position point using different filters to extract the data of the position point, so that each position point can determine an optimal designated filter to extract the data features of the position point; it is also possible to extract features of the data of the position point separately through each filter, and determine the designated filter of the position point by calculating the loss value; finally, the feature data of the position point is extracted through the designated filter of the position point, so that different data can use different filters to extract features.

[0055] The embodiment of the present invention provides a visual task processing method, which extracts feature data of an image to be processed through a preset network model; wherein the network model includes at least one layer of convolutional network; a data matrix including multiple position points is input into the convolutional network; for each position point, a designated filter of the position point is determined according to the data at the position point; and the feature data of the position point is extracted through the designated filter. In this method, for each position point in the data matrix input into the convolutional network, a designated filter of the position point is determined according to the data at the position point, and then the feature data at the position point is extracted based on the designated filter. This method uses different filters to extract features for different data without increasing the number of filters, so that the extracted features are more effective, thereby improving the efficiency of feature extraction, which is conducive to improving the accuracy and other performance of the network model as a whole.

[0056] Furthermore, the convolutional network includes a first sub-network and a second sub-network; wherein the first sub-network is used to: generate a plurality of filters according to the data matrix;

[0057] The above data matrix can be the data matrix output by the previous convolutional network, or it can be the initial image input to the network model; the number and type of the above filters are usually generated according to the input data matrix; in addition, the number of filters in a convolutional network can be the same as the number of channels output by the convolutional network; for example, if the convolutional network requires n channels of feature data to be output, n types of filters can be generated according to the data matrix through the first sub-network. The generated multiple filters are not shared with each other in the spatial dimension of the entire network model, that is, for the input data matrix, the filters used at different positions in the data matrix can be different.

[0058] The second sub-network is used to: for each position point in the data matrix, select a specified filter from a plurality of filters according to the data at the position point, so as to extract the feature data of the position point through the specified filter.

[0059] Usually, for each position point in the data matrix, a standard or ordinary convolutional network usually extracts the feature data of the position point through all the filters in the network; in this embodiment, using the second subnetwork, the classification function in machine learning, such as Logistic, softmax, argmax and other functions, can be used to use the data of each position point in the data matrix to calculate the result data of each filter selected to extract features at each position point. For example, the effect of using each filter to extract features at each position point, the probability of use and other data can be calculated. According to the calculated result data, the specified filter corresponding to each position point can be obtained, and the feature data of the position point can be extracted through the specified filter, so that different data can use different filters to extract features.

[0060] Embodiment three:

[0061] This embodiment provides a flowchart of an application method of the first sub-network in a visual task processing method, and the method is implemented on the basis of the above embodiment. This embodiment focuses on describing the specific application method of the first sub-network. Figure 3 As shown, the method comprises the following steps:

[0062] Step S302, performing an average pooling operation on the data matrix to obtain a pooling result; wherein the scale of the pooling result matches the scale of the filter to be generated;

[0063] The above average pooling operation is to calculate the mean of the data of each position point included in the data matrix; the scale of the filter to be generated can be set according to the characteristics of the data matrix of the image to be processed, for example, it can be a 3×3 filter; in actual implementation, the data matrix is ​​usually averaged according to the scale of the filter to be generated. At the same time, the data matrix can be sampled to obtain a pooling result that matches the scale of the filter to be generated. The pooling result can also be called a feature map. In addition, the above matching with the scale of the filter to be generated can be that the scale of the pooling result is the same as the scale of the filter to be generated, or the scale of the pooling result differs from the scale of the filter to be generated by a certain multiple.

[0064] Preferred implementation manner: The scale of the above-mentioned pooling result is the same as the scale of the filter to be generated; this requires calculating the mean of the data of each position point included in the data matrix, and sampling the data matrix at the same time so that the scale of the pooling result is the same as the scale of the filter to be generated.

[0065] Step S304, performing a first convolution operation on the pooling result to obtain a first convolution result; wherein the number of channels of the first convolution result is associated with the number of types of filters to be generated;

[0066] The first convolution operation is to perform convolution calculation on the pooling result through a convolution kernel of a preset scale to obtain a first convolution result; wherein the scale of the first convolution result is usually unchanged; specifically, through the first convolution operation, the number of channels of the pooling result is converted into a number of channels associated with the number of types of filters to be generated; for example, the number of channels of the first convolution result is the same as the number of types of filters to be generated, or the number of channels of the first convolution result is a multiple of the number of types of filters to be generated, or the number of channels of the first convolution result is a power of the number of types of filters to be generated, etc.; the convolution kernel of the preset scale can be a 1×1 convolution kernel.

[0067] Preferred implementation manner: The number of channels of the first convolution result is equal to the square of the number of types of filters to be generated; for example, if the number of types of filters to be generated is m, the number of channels of the first convolution result is m. 2 .

[0068] Step S306, performing a second convolution operation on the first convolution result to obtain a second convolution result; wherein the second convolution result includes multiple filters; the number of channels of each filter is associated with the number of channels of the data matrix and the number of channels of the convolution network output data.

[0069] The second convolution operation can be performed by convolution calculation on the first convolution result through a convolution kernel of a preset scale in a grouped convolution manner to obtain a second convolution result; in actual implementation, the first convolution result can be subjected to a grouped convolution operation to obtain multiple groups of filters, i.e., the multiple filters mentioned above; the number of groups can be associated with the number of filters to be generated, for example, the number of groups is the same as the number of filters to be generated; it should be noted that the grouped convolution method of the second convolution operation is only divided on the depth of the first convolution result, that is, a certain number of channels is a group, and the specific number of filters in each group, that is, the number of channels of each filter, is determined by the number of channels of the data matrix and the number of channels of the convolution layer output data, for example, the number of channels of each filter is equal to the sum of the number of channels of the data matrix and the number of channels of the convolution layer output data, or the number of channels of each filter is equal to the product of the number of channels of the data matrix and the number of channels of the convolution layer output data, or the number of channels of each filter is equal to the difference between the number of channels of the data matrix and the number of channels of the convolution layer output data, etc.

[0070] Preferred implementation method: the number of channels of each filter is equal to the product of the number of channels of the data matrix and the number of channels of the convolution network output data; this requires determining the specific method of group convolution when performing the second convolution operation on the first convolution result, so that the number of channels of each filter obtained is equal to the product of the number of channels of the data matrix and the number of channels of the convolution layer output data.

[0071] In summary, this embodiment takes the scale of the filter to be generated as 3×3, the number as m, the number of channels of the data matrix as C, the number of output channels as O, the scale size as U×V, and the step size as 1 as an example for specific description. Figure 4 A schematic diagram of a filter generation method is shown.

[0072] First, in order to generate m filters, the data matrix is ​​sampled by an average pooling operation (corresponding to the pooling in the figure) to obtain a data matrix of size U×V, which is a 3×3 data matrix with a channel number C, i.e., the pooling result; then, a 1×1 convolution kernel is used to perform the first convolution operation on the pooling result (corresponding to the first conv in the figure), and the data matrix with a channel number C is converted into a data matrix with a channel number m×m, i.e., the first convolution result mentioned above; finally, a 1×1 group convolution (corresponding to the second conv in the figure) is performed, and the number of groups of the group convolution is set to g=m, to obtain a data matrix with a channel number m×O×C and a size of 3×3, i.e., the second convolution result mentioned above; wherein, the second convolution result includes m filters, each of which has a shape of O×C×3×3, i.e., the number of channels of each filter is equal to the product of the channel number C of the data matrix and the channel number O of the output data of the convolution layer. Specifically, the m filters generated by the data matrix through the first subnetwork can be expressed as W0, W1,…, W m-1 .

[0073] In this method, a first subnetwork included in the convolutional network is used to generate multiple filters according to the data matrix; wherein the multiple filters can be used dynamically in the spatial dimension to facilitate multiple position points in the data matrix, select the specified filter of the position point, and extract the feature data of the position point; without increasing the number of filters, different filters are used to extract features for different data, so that the extracted features are more effective, thereby improving the efficiency of feature extraction, which is beneficial to improving the accuracy and other performance of the network model as a whole.

[0074] Embodiment 4:

[0075] This embodiment provides an application method of the second sub-network in a visual task processing method, which is implemented on the basis of the above embodiment. This embodiment focuses on describing the specific application method of the second sub-network. Figure 5 As shown, the method comprises the following steps:

[0076] Step S502, performing a third convolution operation on the data matrix to obtain a third convolution result; wherein the number of channels of the third convolution result is associated with the number of types of the plurality of filters;

[0077] The third convolution operation is to convolve and calculate the data matrix through a convolution kernel of a preset scale to obtain a third convolution result; wherein the scale of the third convolution result is usually unchanged; specifically, through the third convolution operation, the number of channels of the data matrix is ​​converted into a number of channels associated with the number of types of multiple filters; for example, the number of channels of the third convolution result is the same as the number of types of multiple filters, or the number of channels of the third convolution result is a multiple of the number of types of multiple filters, or the number of channels of the third convolution result is a power of the number of types of multiple filters, etc.; the convolution kernel of the preset scale can be a 1×1 convolution kernel.

[0078] A preferred implementation manner: the number of channels of the third convolution result is the same as the number of types of the plurality of filters; for example, if the number of types of the plurality of filters is m, then the number of channels of the third convolution result is m.

[0079] Step S504, for each position point in the third convolution result, multi-classification processing is performed on the data at the position point to obtain a classification result; the classification result includes a probability value of extracting feature data at the position point using each filter;

[0080] Each position point in the third convolution result usually includes the position points of all channels; for example, if the number of channels of the third convolution result is m, then each position point in the third convolution result represents the position point of the data matrix on the m channels; this implementation takes the number of channels of the third convolution result as m as an example for explanation, and usually the data on the m channels at each position point are subjected to multi-classification processing, that is, through processing methods such as classifiers, such as softmax functions, the data on the m channels at each position point are mapped to values ​​of (0, 1), and the cumulative sum of these values ​​is 1 (satisfying the properties of probability), which can usually be understood as probability; wherein, the number of mapping values ​​at each position point is usually the number of channels of the third convolution result m, and each mapping value is the probability value of one of the channels at the position point using one of the filters to extract the corresponding feature data. In addition, each mapping value, that is, the probability value of each filter extracting feature data, is marked with the corresponding filter and the corresponding position coordinates.

[0081] Step S506: determine the filter with the largest probability value as the target filter for the position point.

[0082] Specifically, a parameter-finding function, such as the argmax function, can be used to select the maximum probability value and the corresponding filter identified by the maximum probability value from each probability value at each position, and the filter is determined as the target filter for the position point.

[0083] In summary, this embodiment takes the generated filter with a scale of 3×3, a number of m, a data matrix with a channel number of C, and a scale of U×V as an example for specific description. Figure 6 Another specific schematic diagram of a filter generation method is shown.

[0084] First, a 1×1 convolution kernel is used to perform the third convolution operation (corresponding to conv in the figure) on the data matrix to convert the data matrix with C channels into a data matrix with m channels, which is the third convolution result mentioned above. Then, in the channel dimension of the data matrix, multi-classification processing is performed on the data of each position point (corresponding to softmax in the figure) to obtain a feature map with m channels and a scale of U×V, which is the classification result mentioned above. Each position point in the feature map contains the probability values ​​of using m filters. Finally, the spatial distribution mask is obtained through the argmax operation (corresponding to the U×V rectangular map in the figure). Each position point in the spatial distribution mask represents the subscript of the filter corresponding to the point with the maximum probability value. Specifically, when performing the feature extraction operation, for any position (u,v) in the spatial dimension, the corresponding value on the spatial distribution mask is i=mask. u,v , then the filter used at this position is W i , the corresponding value in the output feature map is: Y u,v =conv(X u,v ,W i ).

[0085] In this method, the second subnetwork included in the convolutional network is used to perform multi-classification processing on the data at each position point in the data matrix, and a classification result including feature data extracted by each filter at the position point is obtained; considering that partially similar information in the spatial dimension can share the same filter, the selection and design of the filter is guided by the learnable spatial distribution according to the probability value in the classification result; this method uses different filters to extract features for different data without increasing the number of filters, so the extracted features are more effective, thereby improving the efficiency of feature extraction, which is beneficial to improving the accuracy and other performance of the network model as a whole.

[0086] This embodiment provides a specific implementation method. Figure 7 The multi-filter convolution diagram shown in the figure, the above input data matrix corresponds to X in the figure, and the feature data corresponding to the output image to be processed corresponds to Y in the figure; according to the data matrix, m filters W0, W1, ..., W are generated through the first sub-network. m-1; Then, according to the data matrix, the second sub-network is used to generate a feature map with m channels. Each position point of the feature map contains the probability values ​​of using m filters. The subscript of the maximum probability value in the channel dimension is taken to obtain the spatial distribution mask. The range of the spatial distribution mask value is [0, m-1]. The value of each position in the mask represents the filter subscript that should be used at that position. For each position in the spatial dimension, there are m filters to choose from. The corresponding filter can be selected according to the value in the spatial distribution mask for convolution operation to obtain the feature data Y corresponding to the image to be processed.

[0087] Embodiment five:

[0088] This embodiment provides a network model training method for visual task processing, such as Figure 8 As shown, the method comprises the following steps:

[0089] Step S802, inputting the sample image into the initial model; the initial model includes at least one layer of convolutional network;

[0090] The above-mentioned sample images can be multiple images containing faces, objects, scenery, and text; the above-mentioned initial model can be a network model such as CNN (Convolutional Neural Networks), RNN (Recurrent Neural Network), DNN (Deep Neural Network), etc.; the network model can usually include a multi-layer convolutional network and can also include multiple filters.

[0091] Step S804, for the convolutional network, generate multiple filters according to the data matrix input to the convolutional network; for each position point of the data matrix, determine the designated filter of the position point from the multiple filters according to the data at the position point; extract the feature data of the position point through the designated filter;

[0092] The above convolutional network can also be called a convolutional layer. Each convolutional layer in the convolutional neural network is composed of several convolutional units, and the parameters of each convolutional unit are optimized through the back propagation algorithm. The function of the convolutional layer is convolution operation, and the purpose of convolution operation is to extract different features of the input. The above data matrix usually refers to the image data of the above sample image, which can be the data matrix output by the previous convolutional network, or the sample image input to the initial model.

[0093] Specifically, when training the forward propagation of the network model, it includes generating a plurality of filters according to the data matrix using the above-mentioned convolutional network; wherein the number of filters is usually pre-set according to the data matrix; using the above-mentioned convolutional network, the classification functions included in machine learning, such as Logistic, softmax, argmax and other functions, can be used to use the data of each position point in the data matrix to calculate the result data of selecting each filter at each position point. For example, the effect value, use probability and other data of each filter used at each position point can be calculated, and the specified filter corresponding to each position point is obtained according to the calculated result data, and the feature data of the position point is extracted through the specified filter.

[0094] Step S806, calculating the loss value based on the preset loss function and the feature data output by the initial model;

[0095] The above-mentioned preset loss function can measure the difference between the feature data output by the initial model and the real data; usually through the loss function, for example, the difference between the feature data output by the initial model and the real data is calculated, and the difference can be a loss value such as mean square error loss, mean absolute error loss, mean absolute percentage error, etc.; wherein the loss function can be a mean square error loss function, a hinge loss function, a cross entropy loss function, a CTC (Connectionist Temporal Classification, connection time series classification) loss function, etc.

[0096] Step S808, iteratively train the initial model according to the loss value to obtain a trained network model.

[0097] The initial model is iteratively trained, and the network model can be trained by cyclic iteration of forward propagation and back propagation algorithms; in the training process of the network model, the forward propagation algorithm is usually used to calculate the loss value, and the calculated loss value is used to reduce the difference between the output data and the actual data through the back propagation algorithm, and the parameters of the network model are adjusted to optimize the network model. For example, if the calculated loss value does not converge, the next sample image is selected for model training, and the forward propagation algorithm is used to continue to execute steps S802, S804 and S806 until the loss value converges to obtain a network model applied to visual task processing; if the loss value converges, the above steps are stopped, and the trained network model is output as a network model applied to visual task processing. Convergence of loss value means that for different sample images, the output loss value is a fixed value, or is very close to a fixed value; for example, for 10 sample images, the loss value is 10, or between 9.8-10.2, and both cases can be considered that the loss value has converged.

[0098] In addition to the convergence of the loss value, the loss value meeting the preset loss threshold can also be used as the end training condition of the network model applied to visual task processing. It should be noted here that the loss threshold is generally a range, for example, the loss threshold can be 0.1-0.5, or less than 30%, etc. As long as the loss value is within the range of the loss threshold, it can be said that the loss value meets the preset loss threshold.

[0099] The embodiment of the present invention provides a network model training method for visual task processing, which uses sample images to train an initial model including at least one layer of convolution network, generates multiple filters for the convolution network according to a data matrix input to the convolution network; for each position point of the data matrix, determines a designated filter for the position point from multiple filters according to the data at the position point; extracts feature data of the position point through the designated filter; calculates a loss value based on a preset loss function and the output feature data; iteratively trains the initial model according to the loss value to obtain a trained network model; the network model trained in this way uses different filters to extract features for different data without increasing the number of filters, so that the extracted features are more effective, thereby improving the feature extraction efficiency, which is beneficial to improving the accuracy and other performance of the network model as a whole.

[0100] Embodiment six:

[0101] This embodiment provides another network model training method for visual task processing, which is implemented on the basis of the above embodiment. This embodiment focuses on the specific implementation process of the step of determining the specified filter of the position point from multiple filters based on the data at the position point (implemented by step S904), and the specific implementation process of the step of iteratively training the initial model according to the loss value to obtain the trained network model (implemented by steps S908-S910); Fig. 9 As shown, the method comprises the following steps:

[0102] Step S902, inputting the sample image into the initial model; the initial model includes at least one layer of convolutional network;

[0103] Step S904, for the convolutional network, generate multiple filters according to the data matrix input to the convolutional network; for each position point of the data matrix, perform multi-classification processing on the data at the position point to obtain a classification result; wherein the classification result includes the probability value of extracting feature data at the position point using each filter; determine the filter with the largest probability value as the target filter of the position point; extract the feature data of the position point through the target filter;

[0104] The above multi-classification processing, specifically, before the classification processing, first needs to convert the number of channels of the data matrix into the number of generated filters; this embodiment takes the generation of m filters as an example to illustrate, and the data on the m channels at each position point can be multi-classified, that is, through the processing methods such as classifiers, such as softmax function, the data on the m channels at each position point are mapped to (0, 1) values, and the cumulative sum of these values ​​is 1, that is, it satisfies the nature of probability, which can usually be understood as probability; wherein, the number of mapping values ​​at each position point is usually the number of channels m of the third convolution result, and each mapping value is the probability value of one of the channels at the position point using one of the filters to extract the corresponding feature data. In addition, each mapping value, that is, the probability value of each filter extracting feature data, is marked with a corresponding filter and a corresponding position coordinate; using a parameter seeking function, such as argmax function, in each probability value at each position, the maximum probability value and the corresponding filter marked by the maximum probability value are selected, and the filter is determined as the target filter of the position point; through the target filter, the feature data of the position point is extracted.

[0105] Step S906, calculating the loss value based on the preset loss function and the feature data output by the initial model;

[0106] Step S908, for the convolutional network, according to the classification result, determining the target filter of each position point when the convolutional network performs back propagation;

[0107] The above-mentioned back propagation generally refers to an effective method for calculating gradients; specifically, after the loss value is calculated, if the loss value does not converge, for the parameters in the convolutional network, the probability value of using each filter to extract feature data at each position point included in the above-mentioned classification result can be used to determine the target filter of each position point when the convolutional network is performing back propagation by selecting the maximum value, linear addition, linear multiplication, permutation and combination, etc.

[0108] The above step of determining the target filter at each position point when the convolutional network performs back propagation according to the classification results includes:

[0109] For each location point, perform the following operations:

[0110] Step A1, for each filter, multiplying the filter by the probability value of extracting feature data at the position point using the filter to obtain a product result;

[0111] Step A2, summing the product results of each filter to obtain the target filter of the position point.

[0112] Specifically, this embodiment takes the number of filters as m as an example for explanation. For each position point of the data matrix, there are m filters and m probability values ​​representing the use of each filter; when the convolution network performs back propagation and executes the convolution operation of the position point, the product results of the m filters and the corresponding probability values ​​are calculated to obtain m product results; the m product results are linearly added to calculate a new filter for the position point, that is, the above-mentioned target filter.

[0113] Step S910, adjusting the network parameters of the convolutional network according to the loss value and the target filter of each position point to obtain a trained network model.

[0114] If the loss value does not converge, the convolutional network is directional propagated according to the loss value and the target filter at each position point, and the network parameters of the convolutional network are adjusted. Back propagation usually uses the differentiation rule in calculus to calculate the derivatives of two or more composite functions, and back propagates the gradient of the output unit back to the input unit. According to the calculated gradient, the learnable parameter optimization model of the network model is adjusted to obtain the trained network model.

[0115] The above steps of adjusting the network parameters of the convolutional network according to the loss value and the target filter at each position point to obtain the trained network model include:

[0116] Step B1, calculating feature data of each position point according to the target filter of each position point;

[0117] In actual implementation, the probability values ​​of the positions (u, v) on the spatial distribution mask can be α0, α1, …, α m-1 , the target filter of convolution calculation is obtained by weighted summation during back propagation, and convolution calculation is performed through the target filter to extract the feature data of each position point. The process can be expressed as: Y u,v =conv(X u,v , α0·W0+α1·W1+…+α m-1 ·W m-1 ).

[0118] Step B2, calculating the gradient of the classification result according to the feature data of each position point;

[0119] Step B3, adjust the network parameters of the convolutional network according to the gradient and loss value of the classification results.

[0120] Specifically, in order to calculate the gradient of the classification result during back propagation, back propagation can be performed in the manner of forward propagation, that is, the target filter for each position point is calculated according to the method of step S908 above, and the feature data of each position point of the data matrix is ​​extracted through the target filter; based on the feature data of each position point, the gradient of the classification result is calculated, that is, the directional derivative of the probability value of each position point in the classification result using each filter to extract feature data at the feature data of the position point; the gradient includes the direction of the gradient, and the direction of the gradient indicates the direction of error expansion, so when updating the weights of the network parameters of the convolutional network, it is necessary to negate it to reduce the error caused by the weights.

[0121] In this method, when training the network model, the target filter of each position point of the convolutional network during back propagation is determined according to the classification result through forward propagation for the convolutional network; the network parameters of the convolutional network are adjusted according to the loss value and the target filter of each position point to obtain the trained network model; considering that some similar information in the spatial dimension can share the same filter, an effective learnable spatial distribution structure is designed to guide the network model of dynamic use of multiple filters in the spatial dimension, and different filters are used to extract features for different data without increasing the number of filters, so that the extracted features are more effective, thereby improving the efficiency of feature extraction, which is beneficial to improving the accuracy and other performance of the network model as a whole.

[0122] Embodiment seven:

[0123] Corresponding to the above method embodiment, see Fig.10 A structural schematic diagram of a visual task processing device is shown, the device comprising:

[0124] The data output module 1010 is used to obtain the image to be processed, input the image to be processed into a preset network model, and output feature data corresponding to the image to be processed;

[0125] The task processing module 1020 is used to complete a preset visual task based on the feature data;

[0126] Among them, the network model includes at least one layer of convolutional network; the data matrix input to the convolutional network includes multiple position points; for each position point, the feature data of the position point is extracted through the specified filter of the position point; the specified filter is determined according to the data at the position point.

[0127] The embodiment of the present invention provides a visual task processing device, which extracts feature data of an image to be processed through a preset network model; wherein the network model includes at least one layer of convolutional network; a data matrix including multiple position points is input into the convolutional network; for each position point, a designated filter of the position point is determined according to the data at the position point; and the feature data of the position point is extracted through the designated filter. In this method, for each position point in the data matrix input into the convolutional network, a designated filter of the position point is determined according to the data at the position point, and then the feature data at the position point is extracted based on the designated filter. This method uses different filters to extract features for different data without increasing the number of filters, so that the extracted features are more effective, thereby improving the efficiency of feature extraction, which is conducive to improving the accuracy and other performance of the network model as a whole.

[0128] Furthermore, the above-mentioned convolutional network includes a first sub-network and a second sub-network; the first sub-network is used to: generate multiple filters according to the data matrix; the second sub-network is used to: for each position point in the data matrix, select a specified filter from multiple filters according to the data at the position point, so as to extract the feature data of the position point through the specified filter.

[0129] Furthermore, the above-mentioned first subnetwork is specifically used to: perform an average pooling operation on the data matrix to obtain a pooling result; wherein the scale of the pooling result matches the scale of the filter to be generated; perform a first convolution operation on the pooling result to obtain a first convolution result; wherein the number of channels of the first convolution result is associated with the number of types of filters to be generated; perform a second convolution operation on the first convolution result to obtain a second convolution result; wherein the second convolution result includes multiple filters; the number of channels of each filter is associated with the number of channels of the data matrix and the number of channels of the convolution network output data.

[0130] Furthermore, the scale of the above pooling result is the same as the scale of the filter to be generated; the number of channels of the first convolution result is equal to the square of the number of types of filters to be generated; the number of channels of each filter is equal to the product of the number of channels of the data matrix and the number of channels of the convolutional network output data.

[0131] Furthermore, the above-mentioned second subnetwork is specifically used to: perform a third convolution operation on the data matrix to obtain a third convolution result; wherein the number of channels of the third convolution result is associated with the number of types of multiple filters; for each position point in the third convolution result, perform multi-classification processing on the data at the position point to obtain a classification result; the classification result includes the probability value of using each filter to extract feature data at the position point; and determine the filter with the largest probability value as the target filter for the position point.

[0132] Furthermore, the number of channels of the third convolution result is the same as the number of types of the multiple filters.

[0133] The visual task processing device provided in the embodiment of the present invention has the same technical features as the visual task processing method provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0134] Embodiment eight:

[0135] Corresponding to the above method embodiment, see Fig.11 A schematic diagram of the structure of a network model training device for visual task processing is shown, the device comprising:

[0136] An input module 1110, used to input a sample image into an initial model; the initial model includes at least one layer of convolutional network;

[0137] The generating module 1120 is used to generate a plurality of filters for the convolutional network according to the data matrix input to the convolutional network; for each position point of the data matrix, determine a designated filter for the position point from the plurality of filters according to the data at the position point; and extract feature data of the position point through the designated filter;

[0138] A calculation module 1130, used to calculate a loss value based on a preset loss function and feature data output by the initial model;

[0139] The training module 1140 is used to iteratively train the initial model according to the loss value to obtain a trained network model.

[0140] The embodiment of the present invention provides a network model training device for visual task processing, which uses sample images to train an initial model including at least one convolution network, generates multiple filters for the convolution network according to a data matrix input to the convolution network; for each position point of the data matrix, determines a designated filter for the position point from multiple filters according to the data at the position point; extracts feature data of the position point through the designated filter; calculates a loss value based on a preset loss function and the output feature data; iteratively trains the initial model according to the loss value to obtain a trained network model; the network model trained in this way uses different filters to extract features for different data without increasing the number of filters, so that the extracted features are more effective, thereby improving the feature extraction efficiency, which is beneficial to improving the accuracy and other performance of the network model as a whole.

[0141] Furthermore, the generation module is used to: perform multi-classification processing on the data at the position point to obtain a classification result; wherein the classification result includes a probability value of extracting feature data at the position point using each filter; and determine the filter with the largest probability value as the target filter for the position point;

[0142] The above training module is used to: for the convolutional network, determine the target filter of each position point when the convolutional network is performing back propagation according to the classification results; adjust the network parameters of the convolutional network according to the loss value and the target filter of each position point to obtain the trained network model.

[0143] Furthermore, the above training module is used to perform the following operations for each position point: for each filter, multiply the filter by the probability value of extracting feature data at the position point using the filter to obtain a product result; sum the product results of each filter to obtain the target filter for the position point.

[0144] Furthermore, the above training module is used to: calculate the feature data of each position point according to the target filter of each position point; calculate the gradient of the classification result according to the feature data of each position point; and adjust the network parameters of the convolutional network according to the gradient and loss value of the classification result.

[0145] The network model training device for visual task processing provided in the embodiment of the present invention has the same technical features as the network model training method for visual task processing provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0146] Embodiment nine:

[0147] An embodiment of the present invention provides an electronic system, which includes: an image acquisition device, a processing device and a storage device; the image acquisition device is used to obtain preview video frames or image data; a computer program is stored on the storage device, and the computer program executes the steps of the above-mentioned visual task processing method or the above-mentioned network model training method applied to visual task processing when the processed device is running.

[0148] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0149] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processing device, the steps of the above-mentioned visual task processing method or the above-mentioned network model training method applied to visual task processing are executed.

[0150] The computer program products of the visual task processing methods, devices, and electronic systems provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0151] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0152] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0153] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0154] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0155] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A visual task processing method, characterized in that: The method comprises: Acquire an image to be processed, input the image to be processed into a preset network model, and output feature data corresponding to the image to be processed; Based on the feature data, complete a preset visual task; The network model includes at least one layer of convolutional network; the data matrix input to the convolutional network includes a plurality of position points; for each of the position points, the feature data of the position point is extracted through a specified filter of the position point; the specified filter is determined according to the data at the position point; The convolution network includes a first subnetwork and a second subnetwork; the first subnetwork is used to generate a plurality of filters according to the data matrix; the second subnetwork is used to select a specified filter from the plurality of filters for each position point in the data matrix according to the data at the position point, so as to extract feature data of the position point through the specified filter; The first subnetwork is specifically used for: performing an average pooling operation on the data matrix to obtain a pooling result; wherein the scale of the pooling result matches the scale of the filter to be generated; performing a first convolution operation on the pooling result to obtain a first convolution result; wherein the number of channels of the first convolution result is associated with the number of types of filters to be generated; performing a second convolution operation on the first convolution result to obtain a second convolution result; wherein the second convolution result includes multiple filters; the number of channels of each filter is associated with the number of channels of the data matrix and the number of channels of the convolution network output data.

2. The method according to claim 1, characterized in that The scale of the pooling result is the same as the scale of the filter to be generated; The number of channels of the first convolution result is equal to the square of the number of types of filters to be generated; The number of channels of each filter is equal to the product of the number of channels of the data matrix and the number of channels of the convolutional network output data.

3. The method according to claim 1, characterized in that The second sub-network is specifically used for: Performing a third convolution operation on the data matrix to obtain a third convolution result; wherein the number of channels of the third convolution result is associated with the number of types of the plurality of filters; For each position point in the third convolution result, multi-classification processing is performed on the data at the position point to obtain a classification result; the classification result includes a probability value of extracting feature data at the position point using each filter; The filter with the largest probability value is determined as the target filter for the position point.

4. The method according to claim 3, characterized in that The number of channels of the third convolution result is the same as the number of types of the multiple filters.

5. A network model training method for visual task processing, characterized in that: The method comprises: Inputting a sample image into an initial model; the initial model includes at least one layer of convolutional network; For the convolutional network, a plurality of filters are generated according to a data matrix input to the convolutional network; for each position point of the data matrix, a designated filter of the position point is determined from the plurality of filters according to the data at the position point; and feature data of the position point is extracted through the designated filter; Calculating a loss value based on a preset loss function and the feature data output by the initial model; Iteratively train the initial model according to the loss value to obtain a trained network model; The step of determining a designated filter for the location point from the plurality of filters according to the data at the location point comprises: performing multi-classification processing on the data at the location point to obtain a classification result; wherein the classification result comprises a probability value of extracting feature data at the location point using each of the filters; and determining the filter with the largest probability value as the target filter for the location point; The step of iteratively training the initial model according to the loss value to obtain the trained network model includes: for the convolutional network, determining the target filter of each position point when the convolutional network performs back propagation according to the classification result; adjusting the network parameters of the convolutional network according to the loss value and the target filter of each position point to obtain the trained network model.

6. The method according to claim 5, characterized in that The step of determining, according to the classification result, a target filter for each position point when the convolutional network performs back propagation includes: For each of the position points, perform the following operations: For each of the filters, multiply the filter by the probability value of extracting feature data at the position point using the filter to obtain a product result; The product results of each filter are summed to obtain the target filter for the position point.

7. The method according to claim 5, characterized in that The step of adjusting the network parameters of the convolutional network according to the loss value and the target filter of each position point comprises: Calculating feature data of each position point according to the target filter of each position point; Calculating the gradient of the classification result according to the feature data of each position point; According to the gradient of the classification result and the loss value, the network parameters of the convolutional network are adjusted.

8. A visual task processing device, characterized in that: The device comprises: A data output module is used to obtain an image to be processed, input the image to be processed into a preset network model, and output feature data corresponding to the image to be processed; A task processing module, used to complete a preset visual task based on the feature data; The network model includes at least one layer of convolutional network; the data matrix input to the convolutional network includes a plurality of position points; for each of the position points, the feature data of the position point is extracted through a specified filter of the position point; the specified filter is determined according to the data at the position point; The convolution network includes a first subnetwork and a second subnetwork; the first subnetwork is used to generate a plurality of filters according to the data matrix; the second subnetwork is used to select a specified filter from the plurality of filters for each position point in the data matrix according to the data at the position point, so as to extract feature data of the position point through the specified filter; The first subnetwork is specifically used for: performing an average pooling operation on the data matrix to obtain a pooling result; wherein the scale of the pooling result matches the scale of the filter to be generated; performing a first convolution operation on the pooling result to obtain a first convolution result; wherein the number of channels of the first convolution result is associated with the number of types of filters to be generated; performing a second convolution operation on the first convolution result to obtain a second convolution result; wherein the second convolution result includes multiple filters; the number of channels of each filter is associated with the number of channels of the data matrix and the number of channels of the convolution network output data.

9. A network model training device for visual task processing, characterized in that: The device comprises: An input module, used to input a sample image into an initial model; the initial model includes at least one layer of convolutional network; A generation module, for generating a plurality of filters according to a data matrix input to the convolutional network; for each position point of the data matrix, determining a designated filter of the position point from the plurality of filters according to the data at the position point; and extracting feature data of the position point through the designated filter; A calculation module, used to calculate a loss value based on a preset loss function and feature data output by the initial model; A training module, used for iteratively training the initial model according to the loss value to obtain a trained network model; The generating module is used to perform multi-classification processing on the data at the position point to obtain a classification result; wherein the classification result includes the probability value of the position point using each filter to extract feature data; the filter with the largest probability value is determined as the target filter of the position point; The training module is used to determine, for the convolutional network, the target filter of each position point when the convolutional network performs back propagation according to the classification result; adjust the network parameters of the convolutional network according to the loss value and the target filter of each position point to obtain a trained network model.

10. An electronic system, characterized in that: The electronic system includes: a processing device and a storage device; The storage device stores a computer program, which, when executed by the processing device, executes the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 7 are executed.

Citation Information

Patent Citations

  • Method and device for correlating adaptive frames

    CN101527034A

  • Three-dimensional CBCT (cone-beam computed tomography) image denoising method on the basis of improved nonlocal means

    CN105184741A

  • A convolutional neural network accelerator circuit based on a fast filtering algorithm

    CN109948784A

  • Under-sample face recognition method based on multi-dimensional scale transformation network and block weighting method

    CN110210321A

  • Remote sensing image fusion method and system based on multi-scale dynamic convolutional neural network

    CN111080567A