Image feature extraction methods, image segmentation methods, and target detection methods

Abstract

The embodiments of the present application provide an image feature extraction method, an image segmentation method, and a target detection method, which obtains sub-regions by dividing an image to be processed; performs a first expansion operation on the sub-regions to obtain a first region, a second region, and a third region; performs feature extraction on the first region in the length and width directions, on the second region in the width direction, and on the third region in the length direction through a feature extraction layer, and merges the features; alternatively, performs feature region division on the original features according to feature region division parameters, extracts features from the divided features through a feature extraction layer, sets weights for features corresponding to overlapping regions in the extracted features, and merges the features of the extracted features according to the weights; the embodiments of the present application realize feature extraction of images of any size using an MLP model that can only process images of fixed size, thereby improving the applicability of the feature extraction layer.

Description

Technical Field

[0001] The embodiments of the present application relate to, but are not limited to, the field of image processing, and in particular to image feature extraction methods, image segmentation methods, and target detection methods. Background Art

[0002] In the field of computer vision, models based on multi-layer perceptrons (MLP), such as MLP-Mixer, MAXIM, and gMLP, have demonstrated excellent performance by constructing non-attentional deep learning models using only MLP network layers. However, the MLP network layer utilizes a fully connected layer in the token interaction module. The model parameters of the fully connected layer are related to the length and width of the image, making the model unable to process input images of arbitrary sizes, greatly limiting the functionality and application of the MLP model. Furthermore, because the MLP model cannot process images of varying sizes, pre-trained MLP models trained on fixed-size images cannot be directly applied to downstream tasks such as image segmentation, object detection, and instance segmentation. Summary of the Invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The purpose of this application is to solve one of the technical problems existing in the related art to at least a certain extent. The embodiments of this application provide an image feature extraction method, an image segmentation method and a target detection method, which realize feature extraction of images of any size using a feature extraction layer that can only process images of fixed size.

[0005] An embodiment of the first aspect of the present application is a method for extracting image features, comprising:

[0006] Acquire an image to be processed, and input the image to be processed into a preset image processing model to perform a region division operation to obtain subregions, wherein the image processing model includes a feature extraction layer;

[0007] Perform the following steps:

[0008] Performing a first expansion operation on the sub-region to obtain a first region, a second region, and a third region, wherein the size of the first region is equal to a fixed size of the processed image corresponding to the feature extraction layer, the width of the second region is equal to a fixed width of the processed image corresponding to the feature extraction layer, and the length of the third region is equal to a fixed length of the processed image corresponding to the feature extraction layer;

[0009] Extracting features from the first region in the length and width directions using the feature extraction layer to obtain a first feature, extracting features from the second region in the width direction to obtain a second feature, and extracting features from the third region in the length direction to obtain a third feature;

[0010] performing a first feature merging operation based on the first feature, the second feature, and the third feature to obtain a first merged feature;

[0011] Alternatively, perform the following steps:

[0012] Determining feature region division parameters according to the size of the image to be processed and the fixed size of the processed image corresponding to the feature extraction layer;

[0013] Performing feature region division on the original features of the image to be processed according to the feature region division parameters to obtain division features, wherein the size of the division features is equal to the fixed size of the processed image corresponding to the feature extraction layer;

[0014] Performing feature extraction on the segmentation feature by the feature extraction layer to obtain a fourth feature;

[0015] A weight is set for the feature corresponding to the overlapping area in the fourth feature, and a second feature merging operation is performed on the fourth feature according to the weight to obtain a second merged feature.

[0016] According to certain embodiments of the first aspect of the present application, performing a first expansion operation on the sub-region to obtain the first region, the second region, and the third region includes:

[0017] When the size of the image to be processed is smaller than the fixed size of the processed image corresponding to the feature extraction layer, expanding the first sub-region in the sub-region to obtain a first region;

[0018] When the size of the image to be processed is greater than or equal to the fixed size of the processed image corresponding to the feature extraction layer, the second sub-region and the fourth sub-region in the sub-region are expanded to obtain a second region, and the third sub-region and the fourth sub-region in the sub-region are expanded to obtain a third region;

[0019] The first sub-region, the second sub-region, the third sub-region and the fourth sub-region constitute the image to be processed, the size of the image to be processed is (W, H), and the fixed size of the processed image corresponding to the feature extraction layer is (w, h); the size of the first sub-region is (n w w,n h h), n w =[W / w], n h =[H / h]; the size of the second sub-area is (Wnw w,n h h); the size of the third sub-region is (n w w,Hn h h); the size of the fourth sub-area is (Wn w w,Hn h h).

[0020] According to certain embodiments of the first aspect of the present application, the expanding the second sub-region and the fourth sub-region in the sub-region to obtain the second region includes:

[0021] The second sub-region and the fourth sub-region are expanded on a side away from the first sub-region to obtain a first extended region, and the size of the first extended region is (p w ,H),p w =w-(Wn w w);

[0022] The second area is obtained by combining the second sub-area, the fourth sub-area and the first extended area.

[0023] According to certain embodiments of the first aspect of the present application, the expanding the third sub-area and the fourth sub-area in the sub-areas to obtain the third area includes:

[0024] The third sub-region and the fourth sub-region are expanded on a side away from the first sub-region to obtain a second extended region, and the size of the second extended region is (W, p h ),p h =h-(Hn h h);

[0025] The third area is obtained by combining the third sub-area, the fourth sub-area and the second extended area.

[0026] According to certain embodiments of the first aspect of the present application, there are multiple feature extraction layers; for the image to be processed input to the feature extraction layer of the odd layer, a first expansion operation is performed on the image to be processed along a first direction; for the image to be processed input to the feature extraction layer of the even layer, a first expansion operation is performed on the image to be processed along a second direction; the first direction is different from the second direction.

[0027] According to certain embodiments of the first aspect of the present application, performing a first feature merging operation based on the first feature, the second feature, and the third feature to obtain a first merged feature includes:

[0028] cropping the second feature to obtain a first cropping feature corresponding to the second sub-region and the fourth sub-region;

[0029] Clipping the third feature to obtain a second clipping feature corresponding to the third sub-region and the fourth sub-region;

[0030] Obtaining a sub-merged feature corresponding to the fourth sub-region according to the feature corresponding to the fourth sub-region in the second feature and the third feature;

[0031] The sub-merged feature, the first cropping feature, the second cropping feature, and the first feature are merged to obtain a first merged feature.

[0032] According to certain embodiments of the first aspect of the present application, the feature region division parameters include the number of times the feature extraction layer is applied in the width direction, the number of times the feature extraction layer is applied in the length direction, the width of the overlapping region, and the length of the overlapping region; and determining the feature region division parameters based on the size of the image to be processed and the fixed size of the processed image corresponding to the feature extraction layer includes:

[0033] A first sum is obtained by adding a first constant term to a quotient of a width of the image to be processed and a fixed width of the processed image corresponding to the feature extraction layer, and the first sum is rounded to obtain the number of times the feature extraction layer is applied in the width direction;

[0034] A first numerator value is obtained by subtracting the width of the image to be processed from the product of the number of times the feature extraction layer is applied in the width direction and the fixed width of the processed image corresponding to the feature extraction layer, a first denominator value is obtained by the difference between the number of times the feature extraction layer is applied in the width direction and a second constant term, and the quotient of the first numerator value and the first denominator value is rounded to an integer to obtain the width of the overlapping area;

[0035] A second sum is obtained by adding a third constant term to a quotient of the length of the image to be processed and a fixed quotient of the processed image corresponding to the feature extraction layer, and the second sum is rounded to obtain the number of times the feature extraction layer is applied in the length direction;

[0036] A second numerator value is obtained by subtracting the length of the image to be processed from the product of the number of times the feature extraction layer is applied in the length direction and the fixed length of the processed image corresponding to the feature extraction layer. A second denominator value is obtained by the difference between the number of times the feature extraction layer is applied in the length direction and the fourth constant term. The quotient of the second numerator value and the second denominator value is rounded to the integer of the quotient.

[0037] According to certain embodiments of the first aspect of the present application, performing a second feature merging operation on the fourth feature according to the weight to obtain a second merged feature includes:

[0038] linearly summing the feature blocks of the fourth feature in the width direction according to the weight to obtain a fifth feature;

[0039] Linearly summing the feature blocks of the fifth feature in the length direction according to the weight to obtain the second combined feature.

[0040] An embodiment of the second aspect of the present application is an image segmentation method, comprising:

[0041] Obtain the image to be segmented;

[0042] Perform feature extraction on the image to be segmented according to the image feature extraction method described above to obtain a first target feature;

[0043] An image segmentation operation is performed according to the first target feature to obtain a segmented image.

[0044] An embodiment of the third aspect of the present application is a target detection method, comprising:

[0045] Obtain the image to be detected;

[0046] Perform feature extraction on the image to be detected according to the image feature extraction method as described above to obtain a second target feature;

[0047] An image detection operation is performed according to the second target feature to obtain a detected target.

[0048] The above scheme has at least the following beneficial effects: by dividing the image to be processed; when the sub-regions do not overlap, a first expansion operation is performed according to the sub-regions to obtain a first region, a second region and a third region, and the sizes of the first region, the second region and the third region are related to the fixed size of the processed image corresponding to the feature extraction layer, and the feature extraction layer is used to extract features in the length and width directions of the first region, in the width direction of the second region, and in the length direction of the third region, and merge the features; when the sub-regions overlap, the original features of the image to be processed are divided into feature regions according to the feature region division parameters, and the sizes of the divided regions are related to the fixed size of the processed image corresponding to the feature extraction layer, and the divided features are extracted through the feature extraction layer, weights are set for the features corresponding to the overlapping regions in the extracted features, and the extracted features are merged according to the weights; it is realized that the MLP model that can only process images of fixed size can be used to extract features of images of any size, thereby improving the applicability of the feature extraction layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0050] Figure 1 is a step diagram of the image feature extraction method provided in an embodiment of the present application;

[0051] Figure 2 A sub-step diagram of performing a first expansion operation on the sub-region to obtain a first region, a second region, and a third region;

[0052] Figure 3 A sub-step diagram of expanding the second sub-region and the fourth sub-region in the sub-region to obtain the second region;

[0053] Figure 4 A sub-step diagram of expanding the third sub-region and the fourth sub-region in the sub-region to obtain the third region;

[0054] Figure 5 A sub-step diagram of performing a first feature merging operation based on the first feature, the second feature, and the third feature to obtain the first merged feature;

[0055] Figure 6 is a schematic diagram of performing a first expansion operation on the image to be processed in the direction of the upper left corner;

[0056] Figure 7 is a schematic diagram of performing a first expansion operation on the image to be processed in the direction of the lower right corner;

[0057] Figure 8 is a schematic diagram of an image to be processed with overlapping sub-regions;

[0058] Figure 9 is a step diagram of the image segmentation method provided in an embodiment of the present application;

[0059] Figure 10 is a step diagram of the target detection method provided in an embodiment of the present application;

[0060] Figure 11 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0062] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and the like in the specification, claims, or accompanying drawings are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0063] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0064] An embodiment of the present application provides an image feature extraction method.

[0065] Reference Figure 1 , the image feature extraction method includes but is not limited to the following steps:

[0066] Step S100, obtaining an image to be processed and inputting the image to be processed into an image processing model;

[0067] Step S200, performing a region division operation on the image to be processed to obtain sub-regions;

[0068] Perform the following steps:

[0069] Step S310, performing a first expansion operation on the sub-region to obtain a first region, a second region, and a third region;

[0070] Step S320: Using a feature extraction layer, extract features from the first region in the length and width directions to obtain a first feature, extract features from the second region in the width direction to obtain a second feature, and extract features from the third region in the length direction to obtain a third feature.

[0071] Step S330, performing a first feature merging operation based on the first feature, the second feature, and the third feature to obtain a first merged feature;

[0072] Alternatively, perform the following steps:

[0073] Step S410, determining the feature region division parameters according to the size of the image to be processed and the fixed size of the processed image corresponding to the feature extraction layer;

[0074] Step S420, performing feature region division on the original features of the image to be processed according to the feature region division parameters to obtain division features;

[0075] Step S430, extracting the segmented features through the feature extraction layer to obtain a fourth feature;

[0076] Step S440: setting weights for features corresponding to the overlapping areas in the fourth features, and performing a second feature merging operation on the fourth features according to the weights to obtain a second merged feature.

[0077] For step S100, the image to be processed is obtained through an image capturing device such as a camera, a webcam, a video camera, a mobile device, or obtained from the Internet or a database; the image to be processed is input into the image processing model through an input device, an input interface or an input port.

[0078] The size of the image to be processed is defined as (W, H), where W is the width of the image to be processed and H is the length of the image to be processed.

[0079] The image processing model includes multiple feature extraction layers. Specifically, the feature extraction layer is a multilayer perceptron (MLP). The MLP network layer includes an input layer, a hidden layer, and an output layer. The hidden layer nodes of the multilayer perceptron are fully connected, that is, the output of all nodes in the previous layer are connected together and used as the input of the nodes in the next layer.

[0080] The MLP network layer is pre-trained. After pre-training, the MLP network layer determines that the fixed size of the processed image corresponding to the feature extraction layer is (w, h), that is, the fixed size of the image that the feature extraction layer can process is (w, h), w is the fixed width of the processed image corresponding to the feature extraction layer, and h is the fixed length of the processed image corresponding to the feature extraction layer.

[0081] The interactive network layer of the MLP network layer is divided into two core modules: Token Mixing MLP and Channel Mixing MLP. Token Mixing MLP acts on the spatial dimension of image features, while Channel Mixing MLP acts on the channel dimension of image features.

[0082] The MLP network layer is directly applied to the spatial dimension of image features to achieve information interaction between tokens.

[0083] The interaction of tokens is expressed by the following formula: feature w =MLP(w,w);feature h =MLP(h,h).

[0084] Where MLP is a fully connected layer with both input and output dimensions of w (or h); at the same time, tokens in the same row or column share model parameters.

[0085] The aggregation of features by the token interaction network layer of the Token Mixing MLP can be expressed as:

[0086] Where, X in represents the input features, Represents the features output by the MLP network layer in the length direction, Represents the features output by the MLP network layer in the width direction.

[0087] Therefore, the parameters of the MLP network layer are affected by the image size, and the input image size during MLP network layer training is a fixed value, namely w*h.

[0088] In step S200, a region division operation is performed on the image to be processed so that the sub-regions obtained by dividing the image to be processed meet the size suitable for feature extraction by the feature extraction layer; region division is beneficial for the feature extraction layer to extract features from the sub-regions.

[0089] On the one hand, feature extraction is performed through the external padding with cross region method.

[0090] The specific steps of the cross-region external filling method are as follows.

[0091] Reference Figure 2 For step S310, performing a first expansion operation on the sub-regions to obtain the first region, the second region, and the third region includes but is not limited to the following steps:

[0092] Step S311: when the size of the image to be processed is smaller than the fixed size of the processed image corresponding to the feature extraction layer, the first sub-region in the sub-region is expanded to obtain the first region;

[0093] Step S312, when the size of the image to be processed is greater than or equal to the fixed size of the processed image corresponding to the feature extraction layer, the second sub-region and the fourth sub-region in the sub-region are expanded to obtain the second region, and the third sub-region and the fourth sub-region in the sub-region are expanded to obtain the third region.

[0094] The first region is denoted as R A , the second region is represented by R BD , the third region is represented by R CD .

[0095] An image to be processed of size (W, H) is divided into a first sub-region, a second sub-region, a third sub-region, and a fourth sub-region. The first sub-region, the second sub-region, the third sub-region, and the fourth sub-region constitute the image to be processed. The first sub-region is region A, the second sub-region is region B, the third sub-region is region C, and the fourth sub-region is region D.

[0096] The size of the first sub-region is (n w w,n h h), n w =[W / w], n h =[H / h], [] is the rounding symbol.

[0097] For step S311, when the size of the image to be processed is smaller than the fixed size of the processed image corresponding to the feature extraction layer, then n w 、n h If it is less than 1, region A is expanded and filled with zero pixels so that the size of the first region is equal to the fixed size of the processed image corresponding to the feature extraction layer, that is, the size of the first region is [w, h]. Obviously, at this time, the second, third, and fourth sub-regions are empty.

[0098] For step S312, when the size of the image to be processed is greater than or equal to the fixed size of the processed image corresponding to the feature extraction layer, the first sub-region, the second sub-region, the third sub-region and the fourth sub-region are not empty. The size of the second sub-region is (Wn w w,n h h); the size of the third sub-region is (n w w,Hn h h); the size of the fourth sub-region is (Wn w w,Hn h h).

[0099] Reference Figure 3 , expanding the second sub-region and the fourth sub-region in the sub-region to obtain the second region, including but not limited to the following steps:

[0100] Step S3111: Expand the second sub-region and the fourth sub-region on a side away from the first sub-region to obtain a first expanded region;

[0101] Step S3112: The second area is obtained by combining the second sub-area, the fourth sub-area and the first extended area.

[0102] Reference Figure 4 , expanding the third sub-region and the fourth sub-region in the sub-region to obtain a third region, including but not limited to the following steps:

[0103] Step S3121: Expand the third sub-region and the fourth sub-region on a side away from the first sub-region to obtain a second expanded region;

[0104] Step S3122: The third sub-area, the fourth sub-area, and the second extended area are combined to obtain a third area.

[0105] Then, the size of the first extended area is (p w ,H),pw =w-(Wn w w); the size of the second extended area is (W,p h ),p h =h-(Hn h h).

[0106] The width of the second region is equal to the fixed width of the processed image corresponding to the feature extraction layer. Specifically, the size of the second region is (w, H).

[0107] The length of the third region is equal to the fixed length of the processed image corresponding to the feature extraction layer. Specifically, the size of the third region is (W, h).

[0108] Reference Figure 6 , the region is divided from the upper left corner, then the first sub-region is located on the upper left corner of the image to be processed, the second sub-region is located on the right side of the first sub-region, the third sub-region is located below the first sub-region, the fourth sub-region is located on the lower right corner of the image to be processed, the first extended region is located on the right side of the second and fourth sub-regions, and the second extended region is located below the third and fourth sub-regions.

[0109] Reference Figure 7 , the region is divided from the lower right corner, then the first sub-region is located on the lower right corner of the image to be processed, the second sub-region is located on the left side of the first sub-region, the third sub-region is located above the first sub-region, the fourth sub-region is located on the upper left corner of the image to be processed, the first extended region is located on the left side of the second and fourth sub-regions, and the second extended region is located above the third and fourth sub-regions.

[0110] The image processing model has multiple network layers that perform expansion operations. The output of the previous expansion layer will be used as the input of the next expansion layer, and the size of the output of a network layer that performs expansion operations is smaller than the input of the network layer that performs expansion operations.

[0111] In step S320, the token interaction network layer of the MLP network layer extracts features of the first region in the length and width directions, that is, performs feature extraction on the first region. w =MLP(w,w) and feature h =MLP(h,h), get the first feature, the first feature is expressed as F A .

[0112] The token interaction network layer of the MLP network layer extracts the features of the second area in the width direction to obtain the second feature, that is, performs feature extraction on the second area. w =MLP(w,w), the second feature is represented by F BD .

[0113] The token interaction network layer of the MLP network layer extracts the features of the third region in the length direction to obtain the third feature, that is, performs feature extraction on the third region. h =MLP(h,h), the third feature is represented by F CD .

[0114] Reference Figure 5 For step S330, performing a first feature merging operation based on the first feature, the second feature, and the third feature to obtain a first merged feature includes but is not limited to the following steps:

[0115] Step S331, cropping the second feature to obtain a first cropping feature corresponding to the second sub-region and the fourth sub-region;

[0116] Step S332, cropping the third feature to obtain a second cropping feature corresponding to the third sub-region and the fourth sub-region;

[0117] Step S333, obtaining a sub-merged feature corresponding to the fourth sub-region based on the feature corresponding to the fourth sub-region in the second feature and the third feature;

[0118] Step S334 , merging the sub-merged feature, the first cropping feature, the second cropping feature, and the first feature to obtain a first merged feature.

[0119] In this embodiment, the second feature F BD and the third feature F CD Perform cropping to obtain the first cropping features corresponding to the second sub-region and the fourth sub-region and the second cropping features corresponding to the third sub-region and the fourth sub-region. The first cropping features and the second cropping features are consistent with the first feature F A This constitutes an H×W feature, which has the same dimension as the original feature. However, since the fourth sub-region is calculated twice when the features are merged, it is necessary to merge the features corresponding to the fourth sub-region in the second and third features. The following merging calculation is performed on the features of the fourth sub-region: Where, F D is the sub-merged feature corresponding to the fourth sub-region, represents the feature corresponding to the fourth sub-region in the second feature, Indicates the feature corresponding to the fourth sub-region in the third feature.

[0120] When region division is performed in only one direction, the MLP network layer with shared parameters can only act on each independent region, and information exchange between regions cannot be achieved. By performing different feature region divisions in different network layers, overlapping areas exist between the divided regions. Therefore, multiple MLP network layers with shared parameters can act on the features of the overlapping areas in sequence, realizing cross-regional interaction of feature information.

[0121] Specifically, for the image to be processed input to the feature extraction layer of the odd layer, a first expansion operation is performed on the image to be processed according to the first direction; for the image to be processed input to the feature extraction layer of the even layer, a first expansion operation is performed on the image to be processed according to the second direction.

[0122] The first direction is different from the second direction. In this embodiment, the first direction is from the upper left corner, and the second direction is from the lower right corner.

[0123] It is understandable that the MLP network layer contains multiple Token Mixing MLP modules, and the parameters of these modules are not shared. A shared MLP network layer refers to the MLP network layer used for token interaction in the same module, that is, the same MLP network layer is applied to multiple sub-region features.

[0124] On the other hand, feature extraction is performed by sliding overlapping and padding MLP network layers (Sliding MLP with Overlap and Padding).

[0125] The specific steps of the method for sliding overlapping filling MLP network layer are as follows.

[0126] The method of sliding overlapping filling MLP network layer mainly includes: feature area division including overlapping area, feature processing of MLP network layer, and feature area merging.

[0127] Reference Figure 8 , Figure 8 is a schematic diagram of an image to be processed with overlapping sub-regions.

[0128] For feature region division containing overlapping areas, when the width of the image to be processed is smaller than the fixed width of the image that the feature extraction layer can process, or the length of the image to be processed is smaller than the fixed length of the image that the feature extraction layer can process, a padding expansion operation is performed in the corresponding direction. Correspondingly, the width of padding is: wW, and the length of padding is: hH.

[0129] When the width of the image to be processed is greater than or equal to the fixed width of the image that can be processed by the feature extraction layer, or the length of the image to be processed is greater than or equal to the fixed length of the image that can be processed by the feature extraction layer, preliminary feature extraction is performed on the image to be processed through a convolution operation to obtain original features; feature region division parameters are determined according to the size of the image to be processed and the fixed size of the processed image corresponding to the feature extraction layer; and feature region division is performed on the original features of the image to be processed according to the feature region division parameters to obtain divided features.

[0130] The feature region division parameters include the number of times the feature extraction layer is applied in the width direction, the number of times the feature extraction layer is applied in the length direction, the width of the overlapping region, and the length of the overlapping region.

[0131] The number of times the feature extraction layer is applied in the width direction is calculated as follows: the first sum is obtained by adding the quotient of the width of the image to be processed and the fixed width of the processed image corresponding to the feature extraction layer to the first constant term, and the first sum is rounded to obtain the number of times the feature extraction layer is applied in the width direction. Where m w is the number of times the feature extraction layer is applied in the width direction, and 0.5 is the first constant term.

[0132] The width of the overlapping area is calculated as follows: the first numerator is obtained by multiplying the number of times the feature extraction layer is applied in the width direction by the fixed width of the processed image corresponding to the feature extraction layer, and subtracting the width of the image to be processed. The first denominator is obtained by the difference between the number of times the feature extraction layer is applied in the width direction and the second constant term. The quotient of the first numerator and the first denominator is rounded to the integer to obtain the width of the overlapping area. w =[(m w ×wW) / (m w -1.5)]; where l w is the width of the overlapping area, and 1.5 is the second constant term.

[0133] The number of times the feature extraction layer is applied in the length direction is calculated as follows: the second sum is obtained by adding the third constant term to the quotient of the length of the image to be processed and the fixed processed image corresponding to the feature extraction layer, and the second sum is rounded to the integer to obtain the number of times the feature extraction layer is applied in the length direction. Where m h is the number of times the feature extraction layer is applied in the length direction, and 0.5 is the third constant term.

[0134] The length of the overlapping area is calculated as follows: the second numerator is obtained by multiplying the number of times the feature extraction layer is applied in the length direction by the fixed length of the processed image corresponding to the feature extraction layer, and subtracting the length of the image to be processed. The second denominator is obtained by the difference between the number of times the feature extraction layer is applied in the length direction and the fourth constant term. The quotient of the second numerator and the second denominator is rounded to the integer to obtain the length of the overlapping area. h =[(m h ×hH) / (m h -1.5)]; where l h The length of the overlapping area, 1.5 is the fourth constant term.

[0135] According to the above parameters, the original features of the image to be processed with a size of H×W are divided into feature regions to obtain the divided features, which are expressed as F region , the dimension of the partition feature is [m,h,w], where m is the number of partition features.

[0136] For the feature processing of the MLP network layer, since the dimension of the partition feature is [m, h, w], the partition feature can be processed by the MLP network layer. The fourth feature is obtained by extracting the partition feature through the MLP network layer. The fourth feature is expressed as

[0137] For feature region merging, since the feature region division includes overlapping areas, the features need to be merged after being processed by the MLP network layer. When merging features, weights are set for the features of the overlapping areas and linear summation is performed.

[0138] Specifically, performing a second feature merging operation on the fourth feature according to the weight to obtain a second merged feature includes but is not limited to the following steps:

[0139] The weight of the overlapping area of ​​the feature blocks of the fourth feature in the width direction is set to 0.5, and the feature blocks of the fourth feature in the width direction are linearly summed according to the weight to obtain the fifth feature;

[0140] The weight of the overlapping area of ​​the feature blocks of the fifth feature in the length direction is set to 0.5, and the feature blocks of the fifth feature in the length direction are linearly summed according to the weight to obtain the second merged feature.

[0141] The resolution of the second merged feature is H×W, which is consistent with the original feature.

[0142] The shared parameter MLP network layer processes the features of the overlapping regions separately. Since the two overlapping regions belong to different regions, the features obtained after MLP processing are different. To merge the features of the overlapping regions into a single feature, different weights are set and a weighted sum is performed. The features of this region should be relatively consistent with the features of other adjacent regions, that is, the value range of the features before and after merging should be consistent. Since overlapping regions have no obvious specificity, the theoretical optimal weight is 0.5, that is, even weighting.

[0143] In one embodiment, for an image processing model, its MLP network layer can process an image of size 224×224. An image to be processed of size 200×180 is input to the image processing model.

[0144] As can be seen, the 200×180 image size to be processed is smaller than the image size that the MLP network layer can handle. When performing image region segmentation, the 200×180 image can only be expanded to 224×224 through an expansion operation, with an expansion width of 24 and an expansion length of 44.

[0145] When the cross-region external filling method is used for feature extraction, although there is only one valid area of ​​the image, the expansion operation can still be performed from the upper left corner or the upper right corner.

[0146] When the sliding overlapping filling MLP network layer method is used for feature extraction, the expansion operation is performed on both sides of the image to be processed, for example, the length of each side is expanded by 12, and the length of each side is expanded by 22 in the width direction.

[0147] After the MLP network layer is processed, the features are cropped to obtain the output results.

[0148] In one embodiment, for an image processing model, its MLP network layer can process an image of size 224×224. A 420×320 image to be processed is input to the image processing model.

[0149] It can be seen that the 420×320 image to be processed is larger than the image size that the MLP network layer can process.

[0150] When using the cross-region external padding method for feature extraction, the image to be processed is expanded to a size of 448×448. The 448×448 features are then divided into four 224×224 subregions. A shared MLP network layer is applied to each subregion, and the resulting features are merged and cropped to produce a 420×320 image output, thus processing the 420×320 image to be processed.

[0151] When using the sliding overlapping filling MLP network layer method for feature extraction, the image to be processed is directly divided into multiple 224×224 subregions, with some overlapping regions. A shared MLP network layer is applied to each subregion, and the resulting features are merged and cropped to produce a 420×320 image output, thus processing the 420×320 image to be processed.

[0152] An embodiment of the present application provides an image segmentation method.

[0153] Reference Figure 9 , the image segmentation method includes but is not limited to the following steps:

[0154] Step S510, obtaining an image to be segmented;

[0155] Step S520, performing feature extraction on the image to be segmented according to an image feature extraction method to obtain a first target feature;

[0156] Step S530: performing an image segmentation operation according to the first target feature to obtain a segmented image.

[0157] In this embodiment, the public dataset used for the image segmentation task is the ADE20K dataset, which includes 20k training images, 2k verification images, and 3k test images.

[0158] The pre-trained model obtained by the MLP model trained on the ImageNet-1 K dataset for classification tasks is used as the backbone, and the decoder uses the SemanticFPN method to achieve image segmentation.

[0159] Three contrast models are defined:

[0160] The first one is the external filling method, which divides the image to be processed into non-overlapping sub-regions, only starting from the upper left corner for region division, and expanding to the outside of the image.

[0161] The second method is the internal overlapping method, which divides the image to be processed into non-overlapping sub-regions and only expands the area that needs to be expanded toward the inside of the image, that is, intercepts an area of ​​a specified length toward the inside.

[0162] The third method is the internal overlapping area expansion method, which divides the image to be processed into non-overlapping sub-areas. When dividing the areas, two division methods are used: the upper left corner is used as the starting point, and the lower right corner is used as the starting point to achieve cross-area division.

[0163] The segmentation results of each method are shown in Table 1.

[0164] Table 1 Comparison of segmentation results of MLP models processed by different methods on the ADE20K dataset

[0165]

[0166] As can be seen from Table 1, the method of extending the extended area to the outside of the image is better than the method of extending it to the inside of the image, the cross-region method is better than the non-cross-region method, and the method of sliding overlapping filling MLP network layer is better than other methods. It fully considers the information interaction between image blocks and achieves the best segmentation accuracy.

[0167] An embodiment of the present application provides a target detection method.

[0168] Reference Figure 10 ,The target detection method includes but is not limited to the following steps:

[0169] Step S610, obtaining an image to be detected;

[0170] Step S620, performing feature extraction on the image to be detected according to an image feature extraction method to obtain a second target feature;

[0171] Step S630: Perform an image detection operation according to the second target feature to obtain a detection target.

[0172] In this embodiment, a pre-trained model obtained by training an MLP model for classification tasks on the ImageNet-1K dataset is used as the backbone, and the detector adopts the Mask R-CNN method and the Cascade Mask R-CNN method, thereby enabling target detection and recognition.

[0173] The target detection results of Mask R-CNN under different backbone networks can be found in Table 2.

[0174] Table 2 Comparison of target detection results of Mask R-CNN under different backbone networks

[0175]

[0176] The instance segmentation results of Cascade Mask R-CNN under different backbone networks can be referred to Table 3.

[0177] Table 3 Comparison of instance segmentation results of Cascade Mask R-CNN under different backbone networks

[0178]

[0179] As can be seen from Tables 2 and 3, the MLP model using the sliding overlapping filling MLP network layer method can process images of any size and achieve good performance in object detection and instance segmentation tasks, further proving the effectiveness of the sliding overlapping filling MLP network layer method.

[0180] An embodiment of the present application provides an electronic device. Figure 11 The electronic device includes: a memory 220, a processor 210, and a computer program stored in the memory 220 and executable on the processor 210. When the processor 210 executes the computer program, the above-mentioned image feature extraction method, image segmentation method, or target detection method is implemented.

[0181] The electronic device may be any intelligent terminal including a computer.

[0182] In general, for the hardware structure of the electronic device, the processor 210 can be implemented using a general-purpose CPU (Central Processing Unit, central processing unit 210), a microprocessor 210, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0183] The memory 220 can be implemented in the form of a read-only memory 220 (ROM), a static storage device, a dynamic storage device, or a random access memory 220 (RAM). The memory 220 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 220 and is called by the processor 210 to execute the methods of the embodiments of this application.

[0184] The input / output interface is used to realize information input and output.

[0185] The communication interface is used to realize the communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0186] The bus 230 transmits information between the various components of the device (such as the processor 210, memory 220, input / output interface, and communication interface). The processor 210, memory 220, input / output interface, and communication interface are connected to each other within the device through the bus 230.

[0187] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for executing the above-mentioned image feature extraction method, image segmentation method, or object detection method.

[0188] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium. In the above description of this specification, the reference terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" and the like are intended to mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0189] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0190] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0192] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0193] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. Although the embodiments of the present application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and purpose of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0194] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present application, and these equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for extracting image features, characterized in that: Applied to the case of extracting features from images of any size using an MLP model that can only process images of fixed size, the method includes: Acquire an image to be processed, and input the image to be processed into a preset image processing model to perform a region division operation to obtain subregions, wherein the image processing model includes a feature extraction layer; When the sub-regions do not overlap, perform the following steps: Performing a first expansion operation on the sub-region to obtain a first region, a second region, and a third region, wherein the size of the first region is equal to a fixed size of the processed image corresponding to the feature extraction layer, the width of the second region is equal to a fixed width of the processed image corresponding to the feature extraction layer, and the length of the third region is equal to a fixed length of the processed image corresponding to the feature extraction layer; Extracting features from the first region in the length and width directions using the feature extraction layer to obtain a first feature, extracting features from the second region in the width direction to obtain a second feature, and extracting features from the third region in the length direction to obtain a third feature; performing a first feature merging operation based on the first feature, the second feature, and the third feature to obtain a first merged feature; When the sub-regions overlap, perform the following steps: Determining feature region division parameters according to the size of the image to be processed and the fixed size of the processed image corresponding to the feature extraction layer; Performing feature region division on the original features of the image to be processed according to the feature region division parameters to obtain division features, wherein the size of the division features is equal to the fixed size of the processed image corresponding to the feature extraction layer; Performing feature extraction on the segmentation feature by the feature extraction layer to obtain a fourth feature; A weight is set for the feature corresponding to the overlapping area in the fourth feature, and a second feature merging operation is performed on the fourth feature according to the weight to obtain a second merged feature.

2. The image feature extraction method according to claim 1, wherein: The performing a first expansion operation on the sub-region to obtain a first region, a second region, and a third region includes: When the size of the image to be processed is smaller than the fixed size of the processed image corresponding to the feature extraction layer, expanding the first sub-region in the sub-region to obtain a first region; When the size of the image to be processed is greater than or equal to the fixed size of the processed image corresponding to the feature extraction layer, the second sub-region and the fourth sub-region in the sub-region are expanded to obtain a second region, and the third sub-region and the fourth sub-region in the sub-region are expanded to obtain a third region; The first sub-region, the second sub-region, the third sub-region and the fourth sub-region constitute the image to be processed, the size of the image to be processed is (W, H), and the fixed size of the processed image corresponding to the feature extraction layer is (w, h); the size of the first sub-region is (n w w,n h h), n w =[W / w], n h =[H / h]; the size of the second sub-area is (Wn w w,n h h); the size of the third sub-area is (n w w,Hn h h); the size of the fourth sub-area is (Wn w w,Hn h h).

3. The image feature extraction method according to claim 2, wherein: The expanding the second sub-area and the fourth sub-area in the sub-area to obtain the second area includes: The second sub-region and the fourth sub-region are expanded on a side away from the first sub-region to obtain a first extended region, and the size of the first extended region is (p w ,H),p w =w-(Wn w w); The second area is obtained by combining the second sub-area, the fourth sub-area and the first extended area.

4. The image feature extraction method according to claim 2, wherein: The step of expanding the third sub-area and the fourth sub-area in the sub-areas to obtain the third area includes: The third sub-region and the fourth sub-region are expanded on a side away from the first sub-region to obtain a second extended region, and the size of the second extended region is (W, p h ),p h =h-(Hn h h); The third area is obtained by combining the third sub-area, the fourth sub-area and the second extended area.

5. The image feature extraction method according to claim 1, wherein: There are multiple feature extraction layers; for the image to be processed input to the feature extraction layer of the odd layer, a first expansion operation is performed on the image to be processed along a first direction; for the image to be processed input to the feature extraction layer of the even layer, a first expansion operation is performed on the image to be processed along a second direction; the first direction is different from the second direction.

6. The image feature extraction method according to claim 1, wherein: The performing a first feature merging operation according to the first feature, the second feature, and the third feature to obtain a first merged feature includes: cropping the second feature to obtain a first cropping feature corresponding to the second sub-region and the fourth sub-region; Clipping the third feature to obtain a second clipping feature corresponding to the third sub-region and the fourth sub-region; Obtaining a sub-merged feature corresponding to the fourth sub-region according to the feature corresponding to the fourth sub-region in the second feature and the third feature; The sub-merged feature, the first cropping feature, the second cropping feature and the first feature are merged to obtain a first merged feature.

7. The image feature extraction method according to claim 1, wherein: The feature region division parameters include the number of times the feature extraction layer is applied in the width direction, the number of times the feature extraction layer is applied in the length direction, the width of the overlapping region, and the length of the overlapping region; The determining of the feature region division parameters according to the size of the image to be processed and the fixed size of the processed image corresponding to the feature extraction layer includes: A first sum is obtained by adding a first constant term to a quotient of a width of the image to be processed and a fixed width of the processed image corresponding to the feature extraction layer, and the first sum is rounded to obtain the number of times the feature extraction layer is applied in the width direction; A first numerator value is obtained by subtracting the width of the image to be processed from the product of the number of times the feature extraction layer is applied in the width direction and the fixed width of the processed image corresponding to the feature extraction layer, a first denominator value is obtained by the difference between the number of times the feature extraction layer is applied in the width direction and a second constant term, and the quotient of the first numerator value and the first denominator value is rounded to an integer to obtain the width of the overlapping area; A second sum is obtained by adding a third constant term to a quotient of the length of the image to be processed and a fixed quotient of the processed image corresponding to the feature extraction layer, and the second sum is rounded to obtain the number of times the feature extraction layer is applied in the length direction; A second numerator value is obtained by subtracting the length of the image to be processed from the product of the number of times the feature extraction layer is applied in the length direction and the fixed length of the processed image corresponding to the feature extraction layer. A second denominator value is obtained by the difference between the number of times the feature extraction layer is applied in the length direction and the fourth constant term. The quotient of the second numerator value and the second denominator value is rounded to the integer of the quotient.

8. The image feature extraction method according to claim 1, wherein: The performing a second feature merging operation on the fourth feature according to the weight to obtain a second merged feature includes: linearly summing the feature blocks of the fourth feature in the width direction according to the weight to obtain a fifth feature; Linearly summing the feature blocks of the fifth feature in the length direction according to the weight to obtain the second combined feature.

9. An image segmentation method, characterized in that: include: Obtain the image to be segmented; Performing feature extraction on the image to be segmented according to the image feature extraction method according to any one of claims 1 to 8 to obtain a first target feature; An image segmentation operation is performed according to the first target feature to obtain a segmented image.

10. A target detection method, characterized in that: include: Obtain the image to be detected; Performing feature extraction on the image to be detected according to the image feature extraction method according to any one of claims 1 to 8 to obtain a second target feature; An image detection operation is performed according to the second target feature to obtain a detected target.

Images