Image processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to an image processing method, apparatus, electronic device, and storage medium. The method comprises: obtaining a first image feature of an image to be processed; generating one or more filters based on the first image feature and one or more preset filter kernels; obtaining one or more second image features of the image to be processed based on the first image feature and the one or more filters; and determining label information for each pixel in the image to be processed based on the one or more second image features. Embodiments of the present disclosure can improve the accuracy of identifying label information for each pixel in the image to be processed.

Description

[0001] This application is a divisional application of the Chinese patent application filed with the China Patent Office on September 27, 2019, with application number 201910925581.2 and application name “Image processing method, network training method and device, and electronic device”. Technical Field

[0002] The present disclosure relates to the field of computer technology, and in particular to an image processing method, device, electronic device, and storage medium. Background Art

[0003] Semantic segmentation is an important but challenging task in computer vision. Its goal is to assign a class label to each pixel in an image. However, due to the large scale variation of objects, it is unreasonable to predict pixel labels using feature representations at a single scale. Instead, feature representations at multiple scales are required to predict pixel labels.

[0004] Related technologies use filters of different sizes to construct multi-scale feature expressions, but these filters are pre-trained, the filter weights cannot be adjusted, and the adaptability to different images is not good. Summary of the Invention

[0005] The present disclosure proposes an image processing technology solution.

[0006] According to one aspect of the present disclosure, there is provided an image processing method, comprising:

[0007] Acquire a first image feature of an image to be processed;

[0008] generating one or more filters according to the first image feature and one or more preset filter kernels;

[0009] Obtaining one or more second image features of the image to be processed according to the first image feature and the one or more filters;

[0010] determining label information of each pixel in the image to be processed according to the one or more second image features;

[0011] Determining label information of each pixel in the image to be processed according to the one or more second image features includes:

[0012] Determine label information of each pixel in the image to be processed according to the first image feature and the one or more second image features.

[0013] In a possible implementation, the filter is used to construct feature expressions of the image to be processed corresponding to different image scales.

[0014] In a possible implementation, the larger the preset filter kernel is, the larger the feature expression of the image to be processed constructed by the corresponding generated filter is at a larger image scale.

[0015] In a possible implementation manner, the weight of the filter is generated according to the first image feature.

[0016] In a possible implementation, obtaining one or more second image features of the image to be processed according to the first image feature and the one or more filters includes:

[0017] Performing dimensionality reduction processing on the first image feature after the convolution processing to obtain a first image feature after dimensionality reduction;

[0018] Performing deep convolution processing on the first image feature after dimensionality reduction and the one or more filters to obtain one or more third image features;

[0019] Convolution processing is performed on the one or more third image features to obtain one or more second image features of the image to be processed.

[0020] In one possible implementation, the method is implemented by a neural network, and the method further includes: training the neural network according to a preset training set, wherein the training set includes: at least one sample image and original label information of each pixel in the sample image;

[0021] The neural network includes a feature extraction module, at least one dynamic filter generation module and a fusion module, and the training of the neural network according to a preset training set includes:

[0022] Inputting at least one sample image into a feature extraction module to obtain a first image feature;

[0023] Inputting the first image feature into the at least one dynamic filter generation module to obtain one or more filters, and obtaining one or more second image features of the sample image based on the one or more filters and the first image feature;

[0024] Inputting the one or more second image features into the fusion module to determine first label information of each pixel in the sample image;

[0025] Determining a generation loss of the neural network based on original label information of each pixel in the sample image and first label information of each pixel;

[0026] The neural network is trained according to the generation loss.

[0027] In this way, the neural network trained by the image processing method provided by the embodiment of the present disclosure dynamically generates filters that adapt to different image scales according to the image to be processed through the dynamic filter generation module. The weight of the filter is dynamically generated according to the image to be processed, and has better adaptability. As a result, the feature expression of the constructed image to be processed at multiple image scales is more accurate.

[0028] In a possible implementation, the feature extraction module is pre-trained to obtain a network for image feature extraction, and / or the fusion module is a pre-trained neural network for identifying pixel labels.

[0029] According to one aspect of the present disclosure, there is provided an image processing apparatus, comprising:

[0030] An acquisition module, configured to acquire a first image feature of an image to be processed;

[0031] a generating module, configured to generate one or more filters according to the first image feature and one or more preset filter kernels;

[0032] a processing module, configured to obtain one or more second image features of the image to be processed based on the first image features and the one or more filters;

[0033] A determination module is used to determine label information of each pixel in the image to be processed based on the first image feature and the one or more second image features.

[0034] In a possible implementation, the filter is used to construct feature expressions of the image to be processed corresponding to different image scales.

[0035] In a possible implementation, the larger the preset filter kernel is, the larger the feature expression of the image to be processed constructed by the corresponding generated filter is at a larger image scale.

[0036] In a possible implementation manner, the weight of the filter is generated according to the first image feature.

[0037] In a possible implementation, the processing module is further configured to:

[0038] Performing dimensionality reduction processing on the first image feature after the convolution processing to obtain a first image feature after dimensionality reduction;

[0039] Performing deep convolution processing on the first image feature after dimensionality reduction and the one or more filters to obtain one or more third image features;

[0040] Convolution processing is performed on the one or more third image features to obtain one or more second image features of the image to be processed.

[0041] In one possible implementation, the device includes a neural network training module,

[0042] The neural network training module is used to:

[0043] Training the neural network according to a preset training set, wherein the training set includes: at least one sample image and original label information of each pixel in the sample image;

[0044] The neural network includes a feature extraction module, at least one dynamic filter generation module and a fusion module, and the neural network training module is used to:

[0045] Inputting at least one sample image into a feature extraction module to obtain a first image feature;

[0046] Inputting the first image feature into the at least one dynamic filter generation module to obtain one or more filters, and obtaining one or more second image features of the sample image based on the one or more filters and the first image feature;

[0047] Inputting the one or more second image features into the fusion module to determine first label information of each pixel in the sample image;

[0048] Determining a generation loss of the neural network based on original label information of each pixel in the sample image and first label information of each pixel;

[0049] The neural network is trained according to the generation loss.

[0050] In a possible implementation, the feature extraction module is pre-trained to obtain a network for image feature extraction, and / or the fusion module is a pre-trained neural network for identifying pixel labels.

[0051] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0052] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.

[0053] In this way, after obtaining the first image feature of the image to be processed, one or more filters with different convolution kernel sizes can be generated according to the first image feature and one or more preset filter kernels, and one or more second image features of the image to be processed can be obtained according to the one or more filters and the first image feature to construct a feature expression of the image to be processed at multiple image scales, and then determine the label information of each pixel in the image to be processed according to the one or more second image features. The image processing method and device disclosed in the embodiment of the present disclosure, and the electronic device can dynamically generate filters that adapt to different image scales according to the image to be processed. The weight of the filter is dynamically generated according to the image to be processed, and has better adaptability, so that the feature expression of the image to be processed at multiple image scales is more accurate, and the label information of each pixel in the image to be processed is more accurate.

[0054] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0056] Figure 1 A flowchart showing an image processing method according to an embodiment of the present disclosure is shown;

[0057] Figure 2 A flowchart showing an image processing method according to an embodiment of the present disclosure is shown;

[0058] Figure 3 A schematic diagram showing the structure of a neural network for image processing in an embodiment of the present disclosure is shown;

[0059] Figure 4 A block diagram showing an image processing apparatus according to an embodiment of the present disclosure;

[0060] Figure 5 A block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown;

[0061] Figure 6 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0062] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0063] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0064] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0065] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0066] Figure 1 A flowchart of an image processing method according to an embodiment of the present disclosure is shown. The image processing method can be executed by a terminal device or other processing device, wherein the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. Other processing devices can be servers or cloud servers, etc. In some possible implementations, the image processing method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0067] like Figure 1 As shown, the method may include:

[0068] In step S11, a first image feature of an image to be processed is obtained.

[0069] For example, a pre-trained convolutional neural network can be used to perform image feature extraction on the image to be processed to obtain the first image feature of the image to be processed. The training process of the convolutional neural network will be described in detail in the embodiment corresponding to the network training method, and will not be repeated here in the embodiment of the present disclosure.

[0070] In step S12, one or more filters are generated according to the first image feature and one or more preset filter kernels.

[0071] For example, the preset filter kernel can be the size of the convolution kernel of the preset filter to be generated, and different preset filter kernels can generate filters of different convolution kernel sizes. The regional context information in the first image feature can be dynamically extracted according to the preset filter kernel, and the filter can be generated according to the regional context information, that is, the weight of the filter is generated according to the first image feature. When there are multiple preset filter kernels, the regional context information in the first image feature can be extracted respectively according to each preset filter kernel, and then multiple filters can be generated according to the extracted multiple regional context information.

[0072] In one possible implementation, different preset filter kernels correspond to different image scales, that is, multiple filters generated according to different preset filter kernels can be used to construct feature expressions of the image to be processed corresponding to different image scales. In this way, filters with different convolution kernel sizes can be generated by different preset filter kernels, and then the feature expressions of the image to be processed at different image scales can be constructed by filters with different convolution kernel sizes.

[0073] Exemplarily, four filters with convolution kernels of 1×1, 3×3, 5×5, and 7×7 can be generated respectively according to the preset filter kernels of 1×1, 3×3, 5×5, and 7×7 and the first image feature. The larger the preset filter kernel is, the larger the image scale at which the corresponding generated filter can construct the feature expression of the image to be processed.

[0074] In step S13, one or more second image features of the image to be processed are obtained according to the first image features and the one or more filters.

[0075] For example, image processing may be performed on the first image features according to the generated one or more filters to construct second image features corresponding to one or more image scales of the image to be processed.

[0076] Illustratively, in the aforementioned example, four filters with convolution kernel sizes of 1×1, 3×3, 5×5, and 7×7 are generated, and image processing is performed on the first image features respectively, so as to construct the second image features of the image to be processed at four different image scales.

[0077] In step S14, label information of each pixel in the image to be processed is determined according to the one or more second image features.

[0078] For example, the feature expressions (second image features) of the image to be processed at different image scales constructed by various filters may be combined to determine the label information of each pixel in the image to be processed.

[0079] In a possible implementation, determining the label information of each pixel in the image to be processed based on the one or more second image features may include:

[0080] Determine label information of each pixel in the image to be processed according to the first image feature and the one or more second image features.

[0081] For example, the first image feature and one or more second image features can be integrated, and the label information of each pixel in the image to be processed can be identified based on the integrated image features to obtain the label information of each pixel in the image to be processed. In this way, a filter that adapts to different image scales is dynamically generated based on the image to be processed. The weight of this filter is dynamically generated based on the image to be processed, which has better adaptability. As a result, the constructed feature expression of the image to be processed at multiple image scales is more accurate, and the label information of each pixel in the image to be processed determined based on the second image features is more accurate.

[0082] In a possible implementation, determining the label information of each pixel in the image to be processed based on the first image feature and the one or more second image features may include:

[0083] integrating the one or more second image features with the first image feature to obtain an integrated image feature;

[0084] The integrated image features are subjected to convolution processing to obtain label information of each pixel in the image to be processed.

[0085] For example, after constructing the feature expression (second image feature) of the image to be processed at multiple image scales by generating multiple filters and first image features, each second image feature can be integrated with the first image feature, and the image features obtained after integration can be convolved (for example: the integrated image features are convolved using a neural network pre-trained for identifying pixel label information) to obtain the label information of each pixel in the image to be processed.

[0086] Exemplarily, the four filters in the aforementioned example respectively construct the second image features of the image to be processed at four image scales. The four second image features can be integrated with the first image features to obtain integrated image features, and the integrated image features are convolved to obtain label information of each pixel in the image to be processed.

[0087] In this way, after obtaining the first image feature of the image to be processed, one or more filters with different convolution kernel sizes can be generated according to the first image feature and one or more preset filters, and one or more second image features of the image to be processed can be obtained according to the one or more filters and the first image feature, so as to construct a feature expression of the image to be processed at multiple image scales, and then determine the label information of each pixel in the image to be processed according to the one or more second image features. The image processing method disclosed in the embodiment of the present disclosure can dynamically generate filters that adapt to different image scales according to the image to be processed. The weight of the filter is dynamically generated according to the image to be processed, and has better adaptability, so that the feature expression of the image to be processed at multiple image scales constructed is more accurate, thereby making the accuracy of the determined label information of each pixel in the image to be processed higher.

[0088] In a possible implementation, generating one or more filters according to the first image feature and one or more preset filter kernels may include:

[0089] Performing pooling processing on the first image features according to the one or more preset filter kernels to obtain one or more context information;

[0090] Convolution processing is performed on the one or more context information to obtain the one or more filters.

[0091] For example, the first image feature can be pooled using a preset filter kernel (e.g., k×k) of the filter to be generated, thereby pooling the first image feature into a feature representation of the preset filter kernel (k×k), and extracting context information for the region (k×k) within the first image feature. When multiple preset filter kernels exist, multiple pieces of context information can be extracted. Each piece of context information is then convolved to obtain one or more filters.

[0092] For example: when the preset filter kernels are 1×1, 3×3, 5×5, and 7×7 respectively, the first image features can be pooled into feature expressions of 1×1, 3×3, 5×5, and 7×7 respectively to obtain context information of 1×1, 3×3, 5×5, and 7×7, and then the context information of 1×1, 3×3, 5×5, and 7×7 are convolved to obtain filters with convolution kernels of 1×1, 3×3, 5×5, and 7×7 respectively.

[0093] In this way, the first image feature expression of the image to be processed of any size can be pooled into a feature expression (context information) of a specific size (preset filter kernel) through a pooling operation. Filters can be dynamically generated according to the context information of the image to be processed, and the generated filters of different sizes can capture the content of multiple image scales and the feature expressions of multiple image scales.

[0094] In a possible implementation, obtaining one or more second image features of the image to be processed based on the first image features and the one or more filters may include:

[0095] Performing dimensionality reduction processing on the first image feature after the convolution processing to obtain a first image feature after dimensionality reduction;

[0096] Performing deep convolution processing on the first image feature after dimensionality reduction and the one or more filters to obtain one or more third image features;

[0097] Convolution processing is performed on the one or more third image features to obtain one or more second image features of the image to be processed.

[0098] For example, a convolution process can be performed on the first image feature, and the convolution-processed first image feature can be subjected to dimensionality reduction processing to obtain a reduced-dimensional first image feature. The reduced-dimensional first image feature can then be subjected to depthwise convolution processing with each filter to obtain one or more third image features. After convolution processing on each third image feature, a feature representation of the image to be processed at one or more image scales is constructed, and one or more second image features of the image to be processed are obtained.

[0099] In this way, a filter adapted to different image scales can be dynamically generated according to the image to be processed. The weight of the filter is dynamically generated according to the image to be processed, and has better adaptability, thereby constructing a more accurate feature expression of the image to be processed at multiple image scales.

[0100] In one possible implementation, the above-mentioned convolution processing on the context information, and / or the first image feature, and / or the third image feature is to convolve the context information, and / or the first image feature, and / or the third image feature through a 1×1 convolution kernel.

[0101] For example, the aforementioned convolution processing on the context information can be convolution processing of the context information through a 1×1 convolution kernel. In this way, the image processing method provided by the embodiment of the present disclosure can construct a filter that adapts to multiple image scales through a 1×1 convolution kernel, and can convolution processing of the first image feature and the third image feature through a 1×1 convolution kernel, thereby reducing the parameters in the calculation process and effectively reducing the amount of calculation.

[0102] In one possible implementation, the above-mentioned image processing method can be implemented by a neural network. The method can also include: training the neural network according to a preset training set, where the training set includes: at least one sample image and original label information of each pixel in the sample image.

[0103] For example, the embodiments of the present disclosure can pre-train a neural network through a training set. The neural network can extract image features of the image to be processed, generate filters corresponding to different image scales based on the extracted image features, and then construct feature expressions of the image to be processed at different image scales based on the filters. By fusing the feature expressions of the image to be processed at different image scales, the label information of each pixel in the image to be processed can be obtained.

[0104] Figure 2 A flowchart of an image processing method according to an embodiment of the present disclosure is shown. In one possible implementation, the neural network may include a feature extraction module, at least one dynamic filter generation module, and a fusion module.

[0105] like Figure 2 As shown, the above-mentioned training of the neural network according to the preset training set may include:

[0106] In step S15, at least one sample image is input into a feature extraction module to obtain a first image feature.

[0107] In step S16, the first image feature is input into the at least one dynamic filter generation module to obtain one or more filters, and one or more second image features of the sample image are obtained based on the one or more filters and the first image feature;

[0108] In step S17, the one or more second image features are input into the fusion module to determine first label information of each pixel in the sample image.

[0109] For example, the feature extraction module can be trained simultaneously with the dynamic filter generation module, or the feature extraction module can be a pre-trained network for image feature extraction. For example, the classification network can be trained through ImageNet (database), and the classification network includes a convolutional neural network for image feature extraction. The classification network is a network for identifying image categories. ImageNet includes sample images and annotated categories of sample images. The sample images can be identified and processed according to the classification network to obtain the category of the sample image, and then the classification network is trained according to the annotated category of the sample image and the category of the sample image, and the convolutional neural network for image feature extraction in the trained classification network is used as the feature extraction module in the embodiment of the present disclosure for image feature extraction.

[0110] At least one sample image can be input into the feature extraction module to perform image feature extraction. The output of the feature extraction module is the first image feature of the sample image. The first image feature is input into at least one dynamic filter generation module to generate at least one filter that adapts to different image scales.

[0111] For example, the dynamic filter generation module can dynamically extract regional context information from the first image feature based on a preset filter kernel and generate a filter based on the regional context information. Multiple dynamic filter generator modules can respectively extract regional context information from the first image feature based on different preset filter kernels, and then generate multiple filters based on the extracted multiple regional context information. The preset filter kernel is a preset convolution kernel size of the filter to be generated.

[0112] In a possible implementation, the dynamic filter generation module may include: a pooling module and a convolution module.

[0113] Inputting the first image feature into the at least one dynamic filter generation module to obtain one or more filters may include:

[0114] The pooling module performs pooling processing on the first image feature according to a preset filter kernel to obtain context information;

[0115] The convolution module performs convolution processing on the context information to obtain a filter.

[0116] For example, the pooling module can pool the first image feature using a preset filter kernel (e.g., k×k) of the filter to be generated, thereby pooling the first image feature into a feature representation of the preset filter kernel (k×k), and extracting regional (k×k) context information from the first image feature. The convolution module performs convolution on each piece of context information to obtain one or more filters.

[0117] In a possible implementation, the convolution module may perform convolution processing on the context information through a 1*1 convolution kernel.

[0118] After at least one filter is generated by at least one dynamic filter generation module, one or more second image features of the sample image can be obtained through each filter and the first image feature.

[0119] For example, a convolution process can be performed on the first image feature, and the convolution-processed first image feature can be subjected to dimensionality reduction processing to obtain a reduced-dimensional first image feature. The reduced-dimensional first image feature can then be subjected to depthwise convolution processing with each filter to obtain one or more third image features. After convolution processing on each third image feature, a feature representation of the image to be processed at one or more image scales is constructed, and one or more second image features of the image to be processed are obtained.

[0120] For example, after constructing the characteristic expression (second image feature) of the sample image at multiple image scales by generating multiple filters and first image features, one or more second image features and the first image features can be input into a fusion module. The fusion module can integrate the one or more second image features and the first image features, and perform convolution processing on the image features obtained after the integration (for example, the fusion module can be a pre-trained neural network for identifying pixel labels) to obtain the first label information of each pixel in the sample image.

[0121] In step S18, the generation loss of the neural network is determined based on the original label information of each pixel in the sample image and the first label information of each pixel.

[0122] In step S19, the neural network is trained according to the generated loss.

[0123] The generation loss of the neural network can be determined based on the original label information of each pixel and the first label information of each pixel, and then the network parameters of the feature extraction module, the dynamic filter generation module, and the fusion module can be adjusted according to the generation loss until the generation loss of the neural network meets the training requirements, for example: the generation loss is less than the loss threshold, which is a pre-set value.

[0124] In this way, the neural network trained by the image processing method provided by the embodiment of the present disclosure dynamically generates filters that adapt to different image scales according to the image to be processed through the dynamic filter generation module. The weight of the filter is dynamically generated according to the image to be processed, and has better adaptability. As a result, the feature expression of the constructed image to be processed at multiple image scales is more accurate.

[0125] Figure 3 A schematic diagram of the structure of a neural network for image processing in an embodiment of the present disclosure is shown. In order to enable those skilled in the art to better understand the embodiment of the present disclosure, the following is a schematic diagram of the structure of a neural network for image processing in an embodiment of the present disclosure. Figure 3 The shown examples illustrate embodiments of the disclosure.

[0126] Reference Figure 3 The neural network for image processing in the embodiment of the present disclosure includes a feature extraction module 302, multiple dynamic convolution modules 303, and a fusion module 306.

[0127] After image features are extracted from the image to be processed 301 by feature extraction 302, first image features 304 of the image to be processed 301 are obtained. First image features 304 are input into multiple dynamic convolution modules 303 to construct feature representations 305 of the image to be processed 301 at different image scales. Multiple dynamic convolution modules 303 include dynamic convolution module 1, dynamic convolution module 2, ..., and dynamic convolution module m, which construct feature representations at different image scales.

[0128] For any dynamic convolution module, it includes a dynamic filter generation module for generating a filter with a specified convolution kernel size (for example: the dynamic filter generation module of dynamic convolution module 1 is used to generate a filter with a convolution kernel size of 1×1, the dynamic filter generation module of dynamic convolution module 2 is used to generate a filter with a convolution kernel size of 3×3,..., the dynamic filter generation module of dynamic convolution module m is used to generate a filter with a convolution kernel size of k×k).

[0129] Taking dynamic convolution module m as an example, after obtaining the first image feature, the first image feature can be pooled by pooling module 3031 to obtain context information of the k×k region of the first image feature. After convolving the context information with a 1×1 convolution kernel, a filter 3033 with a convolution kernel size of k×k is obtained. At the same time, after performing 1×1 convolution on the first image feature, the convolved first image feature is subjected to dimensionality reduction by dimensionality reduction module 3032. After depth convolution with the filter 2033 with a convolution kernel size of k×k, a third image feature is obtained. The third image feature is then subjected to a 1×1 convolution to obtain a feature expression of the image to be processed at a specific image scale: the second image feature 305.

[0130] The operations of dynamic convolution module 1, dynamic convolution module 2 and other dynamic convolution modules are similar to those of dynamic convolution module m, and will not be repeated here in the embodiment of the present disclosure.

[0131] The second image features 305 and the first image features 304 generated by each dynamic convolution module are input into the fusion module 306. The fusion module 306 integrates the second image features 305 and the first image features 304, and performs convolution processing on the integrated image features to obtain label information 307 of each pixel in the image to be processed.

[0132] The embodiments of the present disclosure can generate filters with different convolution kernel sizes through different dynamic convolution modules to capture multi-image scale content and multi-image scale feature expressions, thereby improving the accuracy of the label information of each pixel in the determined image to be processed.

[0133] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0134] In addition, the present disclosure also provides an image processing device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any image processing method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method section and will not be repeated here.

[0135] Figure 4 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown. Figure 4 As shown, the image processing device includes:

[0136] An acquisition module 401 may be used to acquire a first image feature of an image to be processed;

[0137] A generating module 402 may be configured to generate one or more filters according to the first image feature and one or more preset filter kernels;

[0138] The processing module 403 may be configured to obtain one or more second image features of the image to be processed based on the first image features and the one or more filters;

[0139] The determination module 404 may be configured to determine label information of each pixel in the image to be processed according to the one or more second image features.

[0140] In this way, after obtaining the first image feature of the image to be processed, one or more filters with different convolution kernel sizes can be generated according to the first image feature and one or more preset filter kernels, and one or more second image features of the image to be processed can be obtained according to the one or more filters and the first image feature, so as to construct a feature expression of the image to be processed at multiple image scales, and then determine the label information of each pixel in the image to be processed according to the one or more second image features. The image processing device disclosed in the embodiment of the present disclosure can dynamically generate filters that adapt to different image scales according to the image to be processed. The weight of the filter is dynamically generated according to the image to be processed, and has better adaptability, so that the feature expression of the image to be processed at multiple image scales constructed is more accurate, thereby making the accuracy of the determined label information of each pixel in the image to be processed higher.

[0141] In a possible implementation, different preset filter kernels correspond to different image scales.

[0142] In a possible implementation, the generating module is further configured to:

[0143] Performing pooling processing on the first image features according to the one or more preset filter kernels to obtain one or more context information;

[0144] Convolution processing is performed on the one or more context information to obtain the one or more filters.

[0145] In a possible implementation, the processing module is further configured to:

[0146] Performing dimensionality reduction processing on the first image feature after the convolution processing to obtain a first image feature after dimensionality reduction;

[0147] Performing deep convolution processing on the first image feature after dimensionality reduction and the one or more filters to obtain one or more third image features;

[0148] Convolution processing is performed on the one or more third image features to obtain one or more second image features of the image to be processed.

[0149] In a possible implementation, the determining module is further configured to:

[0150] Determine label information of each pixel in the image to be processed according to the first image feature and the one or more second image features.

[0151] In a possible implementation, the determining module is further configured to:

[0152] integrating the one or more second image features with the first image feature to obtain an integrated image feature;

[0153] The integrated image features are subjected to convolution processing to obtain label information of each pixel in the image to be processed.

[0154] In one possible implementation, the convolution processing for the context information, and / or the first image feature, and / or the third image feature is to convolve the context information, and / or the first image feature, and / or the third image feature through a 1×1 convolution kernel.

[0155] In one possible implementation, the device includes a neural network training module,

[0156] The neural network training module is used to:

[0157] The neural network is trained according to a preset training set, wherein the training set includes: at least one sample image and original label information of each pixel in the sample image.

[0158] In a possible implementation, the neural network includes a feature extraction module, at least one dynamic filter generation module, and a fusion module, and the neural network training module is configured to:

[0159] Inputting at least one sample image into a feature extraction module to obtain a first image feature;

[0160] Inputting the first image feature into the at least one dynamic filter generation module to obtain one or more filters, and obtaining one or more second image features of the sample image based on the one or more filters and the first image feature;

[0161] Inputting the one or more second image features into the fusion module to determine first label information of each pixel in the sample image;

[0162] Determining a generation loss of the neural network based on original label information of each pixel in the sample image and first label information of each pixel;

[0163] The neural network is trained according to the generation loss.

[0164] In a possible implementation, the dynamic filter generation module includes: a pooling module and a convolution module;

[0165] The neural network training module is further used to:

[0166] The pooling module performs pooling processing on the first image feature according to a preset filter kernel to obtain context information;

[0167] The convolution module performs convolution processing on the context information to obtain a filter.

[0168] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0169] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0170] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0171] The embodiments of the present disclosure further provide a computer program product, including computer-readable code. When the computer-readable code runs on a device, a processor in the device executes instructions for implementing the image search method provided in any of the above embodiments.

[0172] The embodiments of the present disclosure further provide another computer program product for storing computer-readable instructions, which, when executed, enable a computer to perform the operations of the image search method provided in any of the above embodiments.

[0173] The electronic device may be provided as a terminal, a server, or other forms of devices.

[0174] Figure 5 The block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0175] Reference Figure 5 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0176] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0177] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0178] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0179] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0180] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0181] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0182] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor assembly 814 can also detect changes in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and temperature changes of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0183] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0184] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0185] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions. The computer program instructions can be executed by the processor 820 of the electronic device 800 to perform the above method.

[0186] Figure 6 FIG1 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. Figure 6 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0187] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0188] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0189] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0190] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0191] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0192] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0193] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0194] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0195] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0196] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0197] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0198] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that: include: Acquire a first image feature of an image to be processed; generating one or more filters according to the first image feature and one or more preset filter kernels, wherein the filters are used to construct feature expressions of the image to be processed corresponding to different image scales; Obtaining one or more second image features of the image to be processed according to the first image feature and the one or more filters; Determining label information of each pixel in the image to be processed according to the one or more second image features; Determining label information of each pixel in the image to be processed according to the one or more second image features includes: determining label information of each pixel in the image to be processed according to the first image feature and the one or more second image features; The generating one or more filters according to the first image feature and one or more preset filter kernels specifically includes: extracting regional context information in the first image feature according to the preset filter kernel, and generating the filter according to the regional context information.

2. The method according to claim 1, characterized in that The larger the preset filter kernel is, the larger the image scale of the feature expression of the image to be processed by the corresponding filter structure is.

3. The method according to claim 1, characterized in that The weight of the filter is generated according to the first image feature.

4. The method according to claim 1, wherein The obtaining, based on the first image feature and the one or more filters, one or more second image features of the image to be processed includes: Performing dimensionality reduction processing on the first image feature after the convolution processing to obtain a first image feature after dimensionality reduction; Performing deep convolution processing on the first image feature after dimensionality reduction and the one or more filters to obtain one or more third image features; Convolution processing is performed on the one or more third image features to obtain one or more second image features of the image to be processed.

5. The method according to any one of claims 1 to 4, characterized in that: The method is implemented by a neural network, and the method further includes: training the neural network according to a preset training set, wherein the training set includes: at least one sample image and original label information of each pixel in the sample image; The neural network includes a feature extraction module, at least one dynamic filter generation module and a fusion module, and the training of the neural network according to a preset training set includes: Inputting at least one sample image into a feature extraction module to obtain a first image feature; Inputting the first image feature into the at least one dynamic filter generation module to obtain one or more filters, and obtaining one or more second image features of the sample image based on the one or more filters and the first image feature; Inputting the one or more second image features into the fusion module to determine first label information of each pixel in the sample image; Determining a generation loss of the neural network based on original label information of each pixel in the sample image and first label information of each pixel; The neural network is trained according to the generation loss.

6. The method according to claim 5, characterized in that The feature extraction module is pre-trained to obtain a network for image feature extraction, and / or the fusion module is a pre-trained neural network for identifying pixel labels.

7. An image processing device, characterized in that: include: An acquisition module, configured to acquire a first image feature of an image to be processed; a generating module, configured to generate one or more filters according to the first image feature and one or more preset filter kernels, wherein the filters are used to construct feature expressions corresponding to different image scales of the image to be processed; a processing module, configured to obtain one or more second image features of the image to be processed based on the first image features and the one or more filters; a determination module, configured to determine label information of each pixel in the image to be processed based on the first image feature and the one or more second image features; The generating module is further configured to extract regional context information from the first image feature according to the preset filter kernel, and generate the filter according to the regional context information.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images