Feature processing method, computer device, and storage medium

Abstract

This application discloses a feature processing method, computer device, and storage medium. The method includes: acquiring the original features of a target image; processing the original features sequentially using multiple cascaded feature processing modules to obtain a first image feature corresponding to each feature processing module; each feature processing module includes multiple basic block networks, each basic block network performing convolution processing based on at least two sub-image features with different frequencies in the network input features to output a second image feature, wherein the network input features of the first basic block network of the first feature processing module are the original features, and the second image feature output by the last basic block network of the feature processing module is the first image feature corresponding to the feature processing module; and fusing the first image features corresponding to at least some of the feature processing modules using a feature fusion module to obtain the target features of the target image. This scheme can reduce the number of parameters and computational load in the feature processing process.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a feature processing method, a computer device, and a computer-readable storage medium. Background Technology

[0002] With the development of image processing technology, the requirements for image feature processing are becoming increasingly stringent. Image feature processing is widely used in image saliency processing, image recognition, object detection, object tracking, and image segmentation.

[0003] In existing technologies, feature processing models are typically used to process image features. However, due to the redundancy of the extracted features, the feature processing models have a large number of parameters and computational load, requiring a significant amount of computing resources. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a feature processing method, computer equipment, and storage medium that can reduce the number of parameters and computational load in the feature processing process.

[0005] To address the aforementioned issues, a first aspect of this application provides a feature processing method. This method includes: acquiring original features of a target image; processing the original features sequentially using multiple cascaded feature processing modules to obtain first image features corresponding to each feature processing module; wherein each feature processing module includes multiple basic block networks, each basic block network performs convolution processing based on at least two sub-image features with different frequencies in the network input features, outputting second image features, the network input features of the first basic block network of the first feature processing module being the original features, and the second image features output by the last basic block network of the feature processing module being used as the first image features corresponding to the feature processing module; and fusing at least some of the first image features corresponding to the feature processing modules using a feature fusion module to obtain target features of the target image.

[0006] To address the aforementioned problems, a second aspect of this application provides a computer device comprising a memory and a processor coupled to each other, wherein the memory stores program data and the processor executes the program data to implement any step of the aforementioned feature processing method.

[0007] To address the aforementioned problems, a third aspect of this application provides a computer-readable storage medium storing program data executable by a processor, the program data being used to implement any step of the aforementioned feature processing method.

[0008] The above scheme obtains the original features of the target image and processes them sequentially using multiple cascaded feature processing modules to obtain the first image features corresponding to each feature processing module. Since the multiple basic block networks of each feature processing module perform convolution processing based on at least two sub-image features with different frequencies in the network input features, the second image features can be obtained by performing convolution processing on sub-image features of different frequencies respectively. This reduces redundant features, preserves the original image information of the target image, and reduces the computational load of the module. In addition, the feature fusion module fuses the first image features corresponding to at least some of the feature processing modules to obtain the target features of the target image, which further reduces the computational load of the model. This comprehensive reduction in the number of parameters and computational load in the feature processing process makes it suitable for porting or use in embedded devices or mobile devices. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in this application, the accompanying drawings required in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Among them:

[0010] Figure 1 This is a schematic diagram of the structure of an embodiment of the image processing model of this application;

[0011] Figure 2 This is a flowchart illustrating the first embodiment of the feature processing method of this application;

[0012] Figure 3 This is a schematic diagram of the structure of an embodiment of the feature processing module of this application;

[0013] Figure 4 This application Figure 2 A flowchart illustrating an embodiment of step S12;

[0014] Figure 5 This is a schematic diagram of the structure of an embodiment of the octave convolution kernel of this application;

[0015] Figure 6 This is a schematic diagram of another embodiment of the octave convolution kernel in this application;

[0016] Figure 7 This is a schematic diagram of the structure of an embodiment of the basic block network of this application;

[0017] Figure 8 This is a schematic diagram of another embodiment of the basic block network of this application;

[0018] Figure 9 This application Figure 2A flowchart illustrating an embodiment of step S13;

[0019] Figure 10 This is a schematic diagram of the structure of an embodiment of the feature fusion module of this application;

[0020] Figure 11 This is a flowchart illustrating the second embodiment of the feature processing method of this application;

[0021] Figure 12 This is a schematic diagram of another embodiment of the image processing model of this application;

[0022] Figure 13 This is a schematic diagram of the structure of an embodiment of the computer device of this application;

[0023] Figure 14 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0025] The terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0026] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0027] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0028] This application provides the following embodiments, and each embodiment is described in detail below.

[0029] Please see Figure 1 The image processing model 100 can be used to perform the following feature processing method, wherein the image processing model 100 may include an input module 101, multiple cascaded feature processing modules 102 and feature fusion module 103. The input module 101, the multiple cascaded feature processing modules 102 and feature fusion module 103 are connected sequentially.

[0030] The input module can be used to obtain the original features of the target image.

[0031] Multiple cascaded feature processing modules 102 can be used to process the original features sequentially to obtain the first image feature corresponding to each feature processing module. Each feature processing module includes multiple basic block networks. Each basic block network performs convolution processing based on at least two sub-image features with different frequencies in the network input features, outputting a second image feature. The network input features of the first basic block network of the first feature processing module are the original features, and the second image feature output by the last basic block network of the feature processing module is used as the first image feature corresponding to the feature processing module.

[0032] The feature fusion module 103 can be used to fuse at least a portion of the first image features corresponding to the feature processing module to obtain the target features of the target image.

[0033] The image processing model 100 is used to execute the feature processing method. The specific implementation process of the feature processing method in the following embodiment can be referred to.

[0034] Please see Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the feature processing method of this application. The method may include the following steps:

[0035] S11: Obtain the original features of the target image.

[0036] The target image can be an image or a frame of a video taken of the target object. The target can be a person, object, animal, scene, etc. This application does not restrict the target image or the method of acquiring the target image.

[0037] The original features of the target image are obtained. The original features may be features extracted directly from the target image, or the target image may be used as the original features, or the original features may be features extracted after preprocessing the target image. This application does not limit this.

[0038] In some implementations, the raw features of the target image can be obtained using the input module of the image processing model.

[0039] S12: The original features are processed sequentially using multiple cascaded feature processing modules to obtain the first image feature corresponding to each feature processing module.

[0040] Multiple feature processing modules are connected sequentially and cascaded, meaning that the output of one feature processing module serves as the input of the next. Each feature processing module includes multiple basic block networks (MSBlocks), which are connected sequentially and cascaded, allowing the output of one basic block network to serve as the input of the next.

[0041] In some implementations, each feature processing module contains a different or the same number of basic block networks, and the dimension of the first image feature corresponding to each feature processing module is different.

[0042] In some implementations, each basic block network can be used to perform convolution processing based on at least two sub-image features with different frequencies in the network input features, and output to obtain the second image feature corresponding to the basic block network. Specifically, the network input features of the first basic block network of the first feature processing module are the original features, and the second image feature output by the last basic block network of the feature processing module is used as the first image feature corresponding to the feature processing module.

[0043] For at least two sub-image features with different frequencies, the image frequency is an indicator of the degree of gray-level change in the image; it is the gradient of gray levels in a planar space. For example, a large desert area in an image is a region with slow gray-level changes, corresponding to a very low frequency value; while an edge region with drastic changes in surface properties is a region with drastic gray-level changes, corresponding to a higher frequency value. Different frequency information plays different roles in image structure. Low-frequency information forms the basic gray-level of the image and has a relatively small effect on determining the image structure; mid-frequency information determines the basic structure of the image and forms the main edge structure; high-frequency information forms the edges and details of the image, further enhancing the image content based on mid-frequency information.

[0044] In some implementations, at least two sub-image features include high-frequency image features and low-frequency image features, that is, the high-frequency and low-frequency image features of the network input features. High-frequency image features include high-frequency components in the image, which can refer to areas of sharp changes in image intensity (brightness / grayscale), i.e., edges (contours), primarily measuring image edges and contours. Low-frequency image features include low-frequency components in the image, referring to areas of gentle changes in image intensity (brightness / grayscale), primarily measuring the overall intensity of the entire image.

[0045] Please see Figure 3 As an example, an image processing model may include four cascaded feature processing modules. These modules may each contain 3, 4, 6, or 4 basic block networks, with each module's basic block networks connected sequentially. The input features of the first basic block network in the first feature processing module are the original features of the target image. The output of the first basic block network serves as the input features of the second basic block network. The output of the last basic block network in the feature processing module serves as the first image feature corresponding to that module. Furthermore, the output of the first feature processing module serves as the input to the second feature processing module (or the first basic block network of the second feature processing module), and so on, to obtain the first image feature corresponding to each feature processing module.

[0046] S13: The feature fusion module is used to fuse at least some of the first image features corresponding to the feature processing module to obtain the target features of the target image.

[0047] After obtaining the first image features corresponding to each feature processing module, at least some of the first image features corresponding to the feature processing modules can be fused to fuse multiple first image features of different dimensions to obtain the target features of the target image. This method can reduce the computational load of the image processing model.

[0048] In this embodiment, by acquiring the original features of the target image, multiple cascaded feature processing modules are used to process the original features sequentially to obtain the first image features corresponding to each feature processing module. Since the multiple basic block networks of each feature processing module perform convolution processing based on at least two sub-image features with different frequencies in the network input features, the second image features can be obtained by performing convolution processing on sub-image features of different frequencies respectively. This can reduce redundant features, retain the original image information of the target image, and reduce the computational load of the module. In addition, the feature fusion module is used to fuse the first image features corresponding to at least some of the feature processing modules to obtain the target features of the target image, which can further reduce the computational load of the model. This can comprehensively reduce the number of parameters and the computational load of the feature processing process, making it suitable for porting or use in embedded devices or mobile devices.

[0049] In some embodiments, please refer to Figure 4 This embodiment can further extend step S12 of the above embodiment. Multiple cascaded feature processing modules sequentially process the original features to obtain the first image feature corresponding to each feature processing module. This embodiment includes the following steps:

[0050] S121: For each basic block network of each feature processing module, at least two sub-image features of the network input features are convolved using an octave convolution kernel to obtain octave convolution features.

[0051] In some implementations, each feature processing module includes multiple cascaded basic block networks, each basic block network including an octave convolution kernel, wherein the multiple basic block networks have similar or identical structures.

[0052] In some implementations, at least two sub-image features include high-frequency image features and low-frequency image features.

[0053] Please see Figure 5 As an example, octave convolution kernels include high-frequency convolution kernels W. H→H High-frequency to low-frequency convolution kernel W H →L Low-frequency convolution kernel W L→L and low-frequency to high-frequency convolution kernel W L→H Among them, the high-frequency convolution kernel W H→H It can also be called a high-frequency to high-frequency convolution kernel, or a low-frequency convolution kernel W. L→L This can also be called a low-frequency to low-frequency convolution kernel. The size of an octave convolution kernel can be k×k, where k is an integer greater than 1. C in C out The number of channels in the feature map (high-frequency image features or low-frequency image features) is represented by α.in and α out This controls the number of high-frequency and low-frequency channels to manage the computational load.

[0054] For each basic network block, at least two sub-image features of the network input features are convolved using octave convolution kernels to obtain octave convolution features, as follows:

[0055] Please see Figure 6 The network input features are divided into at least two high-frequency image features F with different frequencies. H and low-frequency image features F L w and h represent feature maps (high-frequency image features F). H or low-frequency image features F L Width and height.

[0056] For high-frequency to high-frequency transitions, utilize high-frequency convolution kernel W H→H For high-frequency image features F H Perform convolution to obtain the first high-frequency convolution feature.

[0057] For low-frequency to low-frequency transitions, use a low-frequency convolution kernel W. L→L For low-frequency image features F L Perform convolution to obtain the first low-frequency convolution feature.

[0058] Since high-frequency and low-frequency image features have different resolutions, and the octave convolution kernel needs to perform updates within the same frequency and exchanges between different frequencies, resolution adjustment is required before the exchange between high and low frequencies. That is, when going from high frequency to low frequency, downsampling operation can be performed on high-frequency image features, and upsampling operation can be performed when going from low frequency to high frequency.

[0059] For high-frequency to low-frequency, for high-frequency image features F H Perform downsampling processing, for example, on high-frequency image features F H After dimensionality reduction through pooling, high-frequency to low-frequency convolutional kernels W can be used. H→L For the downsampled high-frequency image features F H Perform convolution to obtain the second low-frequency convolution feature.

[0060] For low-frequency to high-frequency, for low-frequency image features F L After upsampling, low-frequency to high-frequency convolution kernels W can be used. L→H The upsampled low-frequency image features are convolved to obtain the second high-frequency convolutional features.

[0061] The first high-frequency convolutional feature and the second high-frequency convolutional feature are superimposed and fused to obtain high-frequency convolutional features. The first low-frequency convolutional feature and the second low-frequency convolutional feature are superimposed and fused to obtain low-frequency convolutional features.

[0062] In some implementations, the resolution of low-frequency image features can be adjusted before convolution, for example, by reducing the resolution of the low-frequency image features to half, i.e., the width and height of the low-frequency image features are 0.5w and 0.5h, respectively. This method helps to reduce redundant feature data and is also beneficial for obtaining global information of the target image.

[0063] In some implementations, features can be transferred between each basic block network via two channels, high frequency and low frequency, that is, high frequency convolutional features and low frequency convolutional features are transferred to subsequent basic block networks as octave convolutional features respectively. Alternatively, features can be transferred between each basic block via a single channel, that is, high frequency convolutional features and low frequency convolutional features are fused to obtain octave convolutional features and then transferred to subsequent basic block networks.

[0064] In some implementations, the high-frequency image features and low-frequency image features F of the network input features of the octave convolution kernel of the above-mentioned input basic block network are... H F L The output second image feature can be represented as F = F{F H+1 F L+1 The input to the output of an octave convolution kernel in a basic block network can be represented as follows:

[0065] F H+1 =g(F H W H→H )+upsample(g(F L W L→H ),twenty one)

[0066] F L+1 =g(F L W L→L )+g(pool(F H ,2),W H→L (2)

[0067] Among them, F H+1 F represents high-frequency convolutional features. L+1 This represents low-frequency convolutional features, g(·) represents convolution operation, upsample(·) represents upsampling operation, and pool(·) represents pooling operation, which is also downsampling operation. The ratio of upsampling operation to downsampling operation is 2.

[0068] Let the size of the input feature map (network input features) be C. in*w*h, the kernel size is C in *C out *k 2 The output feature map size is C out *w*h, the channel control factor is set to α. The total computational cost formula for ordinary convolution is:

[0069] CAL conv =C in ×C out ×w×h×k 2 (3)

[0070] The computational complexity of the aforementioned octave convolution kernel in this application is divided into four parts (high-frequency to high-frequency convolution kernel W). H→H High-frequency to low-frequency convolution kernel W H→L Low-frequency to low-frequency convolution kernel W L→L and low-frequency to high-frequency convolution kernel W L→H The formulas are as follows:

[0071]

[0072]

[0073]

[0074]

[0075]

[0076] The above CAL H→H CAL represents the computational complexity from high to high frequencies. L→L CAL represents the computational complexity from low to low frequencies. H→L CAL represents the computational complexity from high to low frequencies. L→H CAL represents the computational complexity from low to high frequencies. oct The formula (8) represents the total computational cost of the four parts of the above octave convolution kernel. It can characterize the correspondence between the total computational cost of the octave convolution kernel and the total computational cost of ordinary convolution. The computational cost of octave convolution can be controlled by adjusting the value of α. When α = 1, the computational cost of octave convolution is 1 / 4 of that of ordinary convolution.

[0077] The above method uses an octave convolution kernel to convolve the high-frequency and low-frequency image features of the network input features to obtain octave convolution features, which can preserve most of the original image information and reduce the computational cost of the model.

[0078] S122: Obtain the second image feature output by the basic block network using octave convolution features; wherein, the network input feature of the next basic block network in the feature processing module is the second image feature output by the previous basic block network.

[0079] Based on the octave convolution features obtained by convolution using the aforementioned octave convolution kernels, the second image features output by the basic block network can be obtained.

[0080] In some implementations, please refer to Figure 7 The basic network can include sequentially connected octave convolution kernels (OctConv), depthwise separable convolution kernels (DepthConv), and atrous convolution kernels (AtrousConv). Input 1 and Input 2 represent the high-frequency and low-frequency image features of the network input features, respectively. The octave convolution kernels are used to convolve the high-frequency and low-frequency image features of the network input features to obtain octave convolution features, which are then passed to the depthwise separable convolution kernels.

[0081] By using depthwise separable convolution kernels to convolve octave convolution features, depthwise convolution features are obtained. Depthwise separable convolution reduces the number of parameters required for convolution calculation by splitting the correlation between the spatial dimension and channel (depth) dimension of octave convolution features, thereby reducing the number of parameters in the model.

[0082] By performing dilated convolution on the depthwise convolution features using a first dilated convolution kernel, the first dilated convolution features can be obtained, which can expand the receptive field and obtain more contextual information. In addition, the dilation rate of the first dilated convolution kernel can be set based on the specific application scenario, and this application does not impose any restrictions on the dilation rate of the first dilated convolution kernel.

[0083] The octave convolution features obtained by octave convolution kernels are superimposed with the first dilated convolution features to obtain the second image features. The second image features can be passed to the next basic block network to obtain the first image features corresponding to the feature processing module. This method can accelerate the convergence speed of the model.

[0084] In some implementations, the basic block network can also provide a local residual structure, which can directly pass the octave convolution features obtained by convolving the high-frequency and low-frequency image features of the network input features with octave convolution kernels to the next basic block network. This method can accelerate the convergence speed of the model.

[0085] In some implementations, the aforementioned basic block network may also perform convolution only on the high-frequency image features of the network input features to obtain the octave convolution features corresponding to the high-frequency image features, and pass the octave convolution features to the subsequent processing of the basic block network or the next basic block network.

[0086] In some implementations, please refer to Figure 8 The basic block network can include sequentially connected octave convolution kernels, depthwise separable convolution kernels, and a first dilated convolution kernel. Input 1 and Input 2 represent the high-frequency image features and low-frequency image features of the network input features, respectively, and Output 1 and Output 2 represent the superimposed high-frequency features and superimposed low-frequency features (i.e., the second image features) of the basic block network output, respectively.

[0087] The high-frequency and low-frequency image features of the network input are convolved using an octave convolution kernel to obtain high-frequency convolution features and low-frequency convolution features, which are then passed to a depth-separable convolution kernel.

[0088] By using depthwise separable convolution kernels to convolve high-frequency convolution features and low-frequency convolution features respectively, high-frequency depthwise convolution features and low-frequency depthwise convolution features are obtained.

[0089] The high-frequency depth convolution features and low-frequency depth convolution features output by the depth separable convolution kernel are convolved using the first dilated convolution kernel to obtain the high-frequency dilated features and low-frequency dilated features.

[0090] The high-frequency convolutional features are superimposed with the high-frequency dilated features output by the first dilated convolutional kernel to obtain superimposed high-frequency features. Similarly, the low-frequency convolutional features are superimposed with the low-frequency dilated features output by the first dilated convolutional kernel to obtain superimposed low-frequency features. The second image features include both superimposed high-frequency and superimposed low-frequency features, which can be passed to the next basic block network, respectively.

[0091] S123: Use the second image feature output by the last basic block network of the feature processing module as the first image feature corresponding to the feature processing module.

[0092] The feature processing module contains multiple basic block networks. The second image feature output by the last basic block network of the feature processing module is used as the first image feature corresponding to the feature processing module. The first image feature can also be passed to the next feature processing module.

[0093] In this embodiment, octave convolution kernels are used to convolve the high-frequency and low-frequency image features of the network input features to obtain octave convolution features. This reduces feature redundancy. By reducing the resolution of the low-frequency feature images, most of the information in the original image can be preserved, while also reducing the computational load of the model and the consumption of storage space and computing power. In addition, depthwise separable convolution kernels are used to convolve the octave convolution features to obtain depthwise convolution features, which reduces the number of model parameters. First dilated convolution kernels are used to dilate the depthwise convolution features to obtain first dilated convolution features, which increases the receptive field of the features.

[0094] In some embodiments, please refer to Figure 9 This embodiment can further extend step S13 of the above embodiment. By using a feature fusion module to fuse at least a portion of the first image features corresponding to the feature processing modules to obtain the target features of the target image, this embodiment includes the following steps:

[0095] S131: Select at least a portion of the first image features corresponding to multiple feature processing modules from the first image features corresponding to the feature processing modules and perform fusion processing to obtain the first fused feature; wherein, at least a portion of the feature processing modules includes the last feature processing module.

[0096] In some embodiments, at least a portion of the first image features corresponding to multiple feature processing modules can be selected from the first image features corresponding to multiple feature processing modules for fusion processing. The at least a portion of the feature processing modules are two or more feature processing modules. This application does not limit the number of the at least a portion of the feature processing modules selected.

[0097] In some implementations, at least a portion of the first image features corresponding to multiple feature processing modules can be arbitrarily selected from the first image features corresponding to multiple feature processing modules for fusion processing, wherein at least a portion of the feature processing modules includes the last feature processing module.

[0098] In some implementations, at least some of the feature processing modules are consecutively ordered feature processing modules in a cascaded setup. For example, the image processing model always contains 4 feature processing modules, and the first image features corresponding to the 2nd, 3rd, and 4th feature processing modules can be selected from the 4 feature processing modules for fusion processing.

[0099] In some implementations, each feature processing module is also connected to a pooling module, which performs pooling operations on the first image features corresponding to each feature processing module to obtain pooled first image features, which are then input into the next feature processing module.

[0100] In some implementations, the dimensions of the first image features corresponding to each feature processing module are different, that is, the resolutions are different. It is necessary to adjust the first image features corresponding to each feature processing module to the same resolution before performing fusion processing. Fusion processing includes overlay processing, stitching processing, etc., and this application does not limit this.

[0101] In some implementations, the resolution of the first image feature of any feature processing module in at least some of the feature processing modules can be used as the target resolution, and the resolution of the first image feature corresponding to other feature processing modules can be adjusted to the same resolution as the target resolution before fusion processing is performed.

[0102] In some implementations, the first feature processing module or any feature processing module in the connection order of at least some feature processing modules can be used as the target feature processing module, and the remaining feature processing modules can be used as other feature processing modules.

[0103] Please see Figure 10 As an example, at least some of the selected feature processing modules are the second, third, and fourth feature processing modules. The dimensions of the first image features corresponding to the second, third, and fourth feature processing modules are (112×112×40), (56×56×80), and (28×28×80), respectively. The second feature processing module (dimension 112×112×40) can be used as the target feature processing module, and the remaining feature processing modules can be used as other feature processing modules.

[0104] The first image features corresponding to other feature processing modules can be upsampled by a preset multiple to obtain upsampled features. The dimension of the upsampled features is the same as the dimension of the first image features corresponding to the target feature processing module. The preset multiple is related to the order of other feature processing modules.

[0105] Please see Figure 10 As an example, the first image feature (dimension 56×56×80) corresponding to the third feature processing module is upsampled by 2 times (2×Upsample) to obtain the upsampled feature corresponding to the third feature processing module; the first image feature (dimension 28×28×80) corresponding to the fourth feature processing module is upsampled by 4 times (4×Upsample) to obtain the upsampled feature corresponding to the fourth feature processing module.

[0106] The first image feature of the target feature processing module is fused with the features of other feature processing modules to obtain the first fused feature.

[0107] S132: Perform dilated convolution based on the first fusion feature to obtain the target feature of the target image.

[0108] Please see Figure 10 The first fused feature can be convolved using a first convolution kernel (1×1Conv) to obtain the second fused feature. For example, the second fused feature after convolution has a dimension of (112×112×40).

[0109] Multiple dilated convolutional kernels with different dilation rates can be used to dilate the second fused feature, resulting in multiple dilated convolutional features. For example, the multiple dilated convolutional kernels with different dilation rates are (DwConv, d=2), (DwConv, d=4), (DwConv, d=8), and (DwConv, d=16), where the dilation rates of the second dilated convolutional kernels are 2, 4, 8, and 10, respectively. This allows us to obtain features corresponding to different receptive fields with different dilation rates, and to extract feature information at different scales.

[0110] Multiple second-dilated convolutional features are fused to obtain the target features of the target image. The obtained target features incorporate overall information on saliency at different scales.

[0111] In this embodiment, since the first image feature corresponding to the first feature processing module (first stage) has a high resolution and a large amount of computation, fusing the first image features extracted by the feature processing modules of the second, third and fourth stages can improve feature processing efficiency and minimize performance loss. This can achieve network lightweighting in sequence, reduce the computation of the model while maintaining good model performance, and facilitate its use on embedded devices.

[0112] In addition, the image processing model mentioned above has a small number of parameters and can detect salient targets in the target image relatively completely. It can meet the requirements of embedded devices for salient target detection, and the number of parameters and computational load are relatively small, making it suitable for porting and use in embedded devices.

[0113] In some embodiments, the above feature processing method can be used to perform saliency processing on the original features of the target image, which will be illustrated by an example below.

[0114] Please see Figure 11 , Figure 11 This is a flowchart illustrating a second embodiment of the feature processing method of this application. The method may include the following steps:

[0115] S21: Obtain the original features of the target image.

[0116] S22: Multiple cascaded feature processing modules are used to process the original features in sequence to obtain the first image feature corresponding to each feature processing module.

[0117] Each feature processing module includes multiple basic block networks. Each basic block network performs convolution processing based on at least two sub-image features with different frequencies in the network input features to output a second image feature. The network input features of the first basic block network of the first feature processing module are the original features, and the second image feature output by the last basic block network of the feature processing module is used as the first image feature corresponding to the feature processing module.

[0118] S23: The feature fusion module is used to fuse at least some of the first image features corresponding to the feature processing module to obtain the target features of the target image.

[0119] The specific implementation process of steps S21 to S23 above can be referred to the specific implementation process of steps S11 to S13 in the above embodiment, and will not be repeated here.

[0120] Please see Figure 12 As an example, an image processing model may include a four-stage feature processing module, a feature fusion module, a fully connected layer, and a second convolutional kernel. The four-stage feature processing module may contain 3, 4, 6, and 4 basic network blocks, respectively. These cascaded feature processing modules sequentially process the original features, yielding the first image features corresponding to each of the four stages.

[0121] In some implementations, each feature processing module is also connected to a pooling module, which performs pooling operations on the first image features corresponding to each feature processing module to obtain the pooled first image features, which are then input into the next feature processing module.

[0122] The steps described above in the feature processing method can be used to perform saliency processing on the original features of the target image and extract the overall information of salient targets at different scales in the target image.

[0123] In some implementations, as an example, the specific structural information of the multiple feature processing modules mentioned above can be found in the following table.

[0124] Table 1. Structural Information of the Feature Extraction Module

[0125]

[0126] In the table above, Input layer represents the input module (or input layer) used to acquire the original features of the target image, MSBlock represents the basic block network, AvgPool represents the average pooling operation, and Pool represents the pooling operation. Feature map size branch 1 represents the branch for high-frequency image features, and feature map size branch 2 represents the branch for low-frequency image features.

[0127] Stride: The sampling interval of the convolution kernel through the input feature map can reduce the number of input parameters and reduce the amount of computation.

[0128] Padding: Add a certain number of rows and columns to each side of the input feature map so that the length and width of the output feature map are equal to the length and width of the input feature map, and so that each square of the input feature map can serve as the center of the convolution window.

[0129] The image processing model includes four stages of feature processing modules as shown in the table above. The first stage feature processing module (Stage1) contains three basic block networks (MSBlock*3), with a convolution kernel size of 3. The size of high-frequency image features is 224*224*10, and the size of low-frequency image features is 112*112*10.

[0130] The image processing model also includes a pooling module. Each stage's feature processing module is connected to a pooling module, which is used to perform pooling processing on the first image features output by each stage's feature processing module before inputting them into the next stage's feature processing module.

[0131] The pooling module (Pool1) is used to perform pooling processing on the first image features output by the first-stage feature processing module, and then input them into the second-stage feature processing module.

[0132] The second-stage feature processing modules (Stage2.1, Stage2.2, Stage2.3) can each contain one basic block network (MSBlock), two basic block networks (MSBlock*2), and one basic block network (MSBlock), respectively. After processing the first image features output by the first-stage feature processing modules, the first image features corresponding to the second-stage feature processing modules (Stage2.1, Stage2.2, Stage2.3) are obtained.

[0133] The pooling module (Pool2) is used to perform pooling processing on the first image features output by the second-stage feature processing modules (Stage2.1, Stage2.2, Stage2.3) and then input them into the third-stage feature processing module.

[0134] The third-stage feature processing modules (Stage3.1, Stage3.2, Stage3.3) can each contain one basic block network (MSBlock), four basic block networks (MSBlock*4), and one basic block network (MSBlock), respectively. After processing the first image features output by the second-stage feature processing modules, the first image features corresponding to the third-stage feature processing modules (Stage3.1, Stage3.2, Stage3.3) are obtained.

[0135] The pooling module (Pool3) is used to perform pooling processing on the first image features output by the feature processing modules of the third stage (Stage3.1, Stage3.2, Stage3.3) and then input them into the feature processing module of the fourth stage.

[0136] The fourth-stage feature processing modules (Stage4.1, Stage4.2, Stage4.3) can each contain one basic block network (MSBlock), two basic block networks (MSBlock*2), and one basic block network (MSBlock), respectively. After processing the first image features output by the third-stage feature processing modules, the first image features corresponding to the fourth-stage feature processing modules (Stage4.1, Stage4.2, Stage4.3) are obtained.

[0137] In some applications, basic block networks can perform convolution processing on high-frequency and low-frequency image features of the network input features to obtain the second image features output by the basic block network.

[0138] In some applications, basic block networks can perform convolution processing on high-frequency image features from the network input features to obtain the second image features output by the basic block network.

[0139] To further reduce the computational cost and number of parameters in the model, some basic block networks in the feature processing module can be convolved based on the high-frequency image features of the network input features to obtain the second image features output by the basic block network.

[0140] Since the first stage has a high resolution and a large computational load, in order to improve efficiency and minimize performance loss, a feature fusion module is used to fuse the first image features corresponding to the feature processing modules of the second, third, and fourth stages. The first image features corresponding to the feature processing modules of the third and fourth stages are upsampled by 2x and 4x respectively to improve the resolution to the same level as the first image features corresponding to the feature processing modules of the second stage. The features are then stitched together to obtain the first fused feature. Then, the first convolution kernel (1×1 Conv) is used for convolution to obtain the second fused feature. In order to obtain feature maps under different receptive fields, the second fused feature is convolved with dilatation rates of 2, 4, 8, and 16 respectively, and then fused to obtain the target feature. The target feature fuses feature information from multiple scales and different receptive fields, thereby achieving network lightweighting.

[0141] S24: The target features of the target image are processed sequentially using a fully connected layer and a second convolutional kernel to obtain the saliency map of the target image.

[0142] By using a fully connected layer to process the target features of the target image, and then using a second convolutional kernel (1×1Conv) for convolution, a saliency map of the target image can be output.

[0143] The image processing model in this embodiment has a small number of model parameters and can detect salient targets in the target image relatively completely. It can meet the requirements of embedded devices for salient target detection, and the number of parameters and computational load are relatively small, making it suitable for porting and use in embedded devices.

[0144] The specific implementation of this embodiment can be referred to the implementation process of the above embodiments, and will not be repeated here.

[0145] Regarding the above embodiments, this application provides a computer device; please refer to [link / reference]. Figure 13 , Figure 13 This is a schematic diagram of the structure of a computer device according to an embodiment of the present application. The computer device 30 includes a memory 31 and a processor 32, wherein the memory 31 and the processor 32 are coupled to each other. The memory 31 stores program data, and the processor 32 is used to execute the program data to implement the steps in any of the above-described feature processing methods.

[0146] In this embodiment, processor 32 can also be referred to as CPU (Central Processing Unit). Processor 32 may be an integrated circuit chip with signal processing capabilities. Processor 32 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor, or processor 32 can be any conventional processor.

[0147] The specific implementation of this embodiment can be referred to the implementation process of the above embodiments, and will not be repeated here.

[0148] The methods described in the above embodiments can be implemented as computer programs; therefore, this application proposes a computer-readable storage medium. Please refer to [link to relevant documentation]. Figure 14 , Figure 14 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 40 stores program data 41 that can be executed by a processor. The program data 41 can be executed by the processor to implement the steps of any embodiment of the feature processing method described above.

[0149] In this embodiment, the computer-readable storage medium 40 can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or a medium that can store program data 41. Alternatively, it can be a server that stores the program data 41. The server can send the stored program data 41 to other devices for execution, or it can run the stored program data 41 itself.

[0150] The specific implementation of this embodiment can be referred to the implementation process of the above embodiments, and will not be repeated here.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0152] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0153] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application.

[0155] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, and thus stored in a computer-readable storage medium for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Therefore, this application is not limited to any particular hardware and software combination.

[0156] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A feature processing method characterized by, The method includes: Obtain the original features of the target image; The original features are processed sequentially using multiple cascaded feature processing modules to obtain a first image feature corresponding to each feature processing module. Each feature processing module includes multiple cascaded basic block networks. Each basic block network performs convolution processing based on at least two sub-image features with different frequencies in the network input features, outputting a second image feature. The network input features of the first basic block network of the first feature processing module are the original features, and the second image feature output by the last basic block network of the feature processing module is used as the first image feature corresponding to that feature processing module. The basic block network includes sequentially connected octave convolution kernels, depthwise separable convolution kernels, and a first dilated convolution kernel. For each basic block network of each feature processing module, the at least two sub-image features of the network input features are convolved using the octave convolution kernel to obtain octave convolution features. The octave convolution features are then convolved using the depthwise separable convolution kernel to obtain depthwise convolution features. The depthwise convolution features are then dilated using the first dilated convolution kernel to obtain first dilated convolution features. The octave convolution features and the first dilated convolution features are then superimposed to obtain the second image features output by the basic block network. The network input features of the subsequent basic block network in the feature processing module are the second image features output by the preceding basic block network. The second image feature output by the last basic block network of the feature processing module is used as the first image feature corresponding to the feature processing module. The feature fusion module is used to fuse at least a portion of the first image features corresponding to the feature processing module to obtain the target features of the target image.

2. The method of claim 1, wherein, The at least two sub-image features include high-frequency image features and low-frequency image features, and the octave convolution kernel includes a high-frequency convolution kernel, a high-frequency to low-frequency convolution kernel, a low-frequency convolution kernel, and a low-frequency to high-frequency convolution kernel; The step of using the octave convolution kernel to convolve at least two sub-image features of the network input features to obtain octave convolution features includes: The high-frequency image features are convolved using the high-frequency convolution kernel to obtain the first high-frequency convolution feature; The low-frequency image features are upsampled, and the upsampled low-frequency image features are convolved using the low-frequency to high-frequency convolution kernel to obtain the second high-frequency convolution features; The low-frequency image features are convolved using the low-frequency convolution kernel to obtain the first low-frequency convolution feature; The high-frequency image features are downsampled, and the high-frequency to low-frequency convolution kernel is used to convolve the downsampled high-frequency image features to obtain the second low-frequency convolution feature. High-frequency convolution features are obtained from the first high-frequency convolution features and the second high-frequency convolution features, and low-frequency convolution features are obtained from the first low-frequency convolution features and the second low-frequency convolution features; The high-frequency convolutional features and the low-frequency convolutional features are used as the octave convolutional features, or the high-frequency convolutional features and the low-frequency convolutional features are fused to obtain the octave convolutional features.

3. The method of claim 1, wherein, The step of fusing at least a portion of the first image features corresponding to the feature processing module using the feature fusion module to obtain the target features of the target image includes: From the first image features corresponding to the plurality of feature processing modules, at least a portion of the first image features corresponding to the feature processing modules are selected for fusion processing to obtain a first fused feature; wherein, the at least a portion of the feature processing modules includes the last feature processing module; Based on the first fusion feature, dilated convolution is performed to obtain the target feature of the target image.

4. The method of claim 3, wherein, At least some of the feature processing modules are consecutively ordered feature processing modules in a cascaded setup; The step of selecting at least a portion of the feature processing modules corresponding to the first image features for fusion processing to obtain the first fused features includes: The first feature processing module in the connection sorting of the at least part of the feature processing modules is taken as the target feature processing module, and the remaining feature processing modules are taken as other feature processing modules. The first image features corresponding to the other feature processing modules are upsampled by a preset multiple to obtain upsampled features; wherein the dimension of the upsampled features is the same as the dimension of the first image features corresponding to the target feature processing module, and the preset multiple is related to the order of the other feature processing modules; The first image feature of the target feature processing module is fused with the features of each of the other feature processing modules to obtain the first fused feature.

5. The method of claim 3, wherein, The step of performing dilated convolution based on the first fusion feature to obtain the target feature of the target image includes: The first fused feature is convolved using the first convolution kernel to obtain the second fused feature; The second fused feature is dilated and convolved using multiple second dilated convolution kernels with different dilation rates, resulting in multiple second dilated convolution features. The multiple second dilated convolutional features are fused to obtain the target features of the target image.

6. The method of claim 1, wherein, The feature processing includes saliency processing of the original features of the target image; After fusing at least a portion of the first image features corresponding to the feature processing module using the feature fusion module to obtain the target features of the target image, the method further includes: The target features of the target image are processed sequentially using a fully connected layer and a second convolutional kernel to obtain the saliency map of the target image.

7. A computer device, characterized in that, The method includes a memory and a processor coupled to each other, the memory storing program data and the processor executing the program data to implement the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The system stores program data that can be executed by a processor, the program data being used to implement the steps of the method according to any one of claims 1 to 6.

Images