Feature extraction method and device, storage medium and electronic equipment

Abstract

The present disclosure provides a feature extraction method and device, a storage medium and an electronic device; and relates to the technical field of artificial intelligence. The method comprises: obtaining input data of a preset neural network according to to-be-recognized data and local data in the to-be-recognized data; and performing fusion feature extraction on the input data by using the preset neural network to obtain target features of the to-be-recognized data. The present disclosure uses a neural network to perform fusion feature extraction on global data and local data, and can improve the feature accuracy of neural network recognition.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a feature extraction method, a feature extraction device, a computer-readable storage medium, and an electronic device. Background Technology

[0002] With the advent of the big data era, neural networks have been applied to various fields of artificial intelligence, such as image recognition and autonomous vehicles.

[0003] For example, in the field of image recognition, image features extracted using neural networks can be used to perform specific tasks in subsequent neural networks, such as face recognition and image segmentation. Improving the accuracy of image feature extraction has always been a highly relevant technical challenge.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a feature extraction method, feature extraction device, computer-readable storage medium, and electronic device, thereby overcoming, to at least a certain extent, the limitation of feature accuracy in neural network recognition caused by related technologies, resulting in low accuracy of neural network recognition.

[0006] According to a first aspect of this disclosure, a feature extraction method is provided, comprising:

[0007] The input data of the preset neural network is obtained based on the data to be identified and local data in the data to be identified;

[0008] The input data is fused and extracted using the preset neural network to obtain the target features of the data to be identified.

[0009] In one exemplary embodiment of this disclosure, the preset neural network includes at least an encoder network and a decoder network. The encoder network includes multiple downsampling layers, each downsampling layer including at least multiple convolutional layers and a pooling layer. The decoder network includes multiple upsampling layers, each upsampling layer including at least a deconvolutional layer and multiple convolutional layers.

[0010] In one exemplary embodiment of this disclosure, the preset neural network further includes a local feature extraction network, the network structure of which is the same as that of the encoder network; the step of obtaining input data for the preset neural network based on the data to be identified and the local data in the data to be identified includes:

[0011] The data to be identified is used as the input data of the encoder network;

[0012] The local data of the data to be identified is used as the input data of the local feature extraction network.

[0013] In one exemplary embodiment of this disclosure, the step of using the preset neural network to perform fusion feature extraction on the input data to obtain the target features of the data to be identified includes:

[0014] The encoder network is used to extract features from the data to be identified, and the first output feature of the encoder network is obtained.

[0015] The local feature extraction network is used to extract features from local data in the data to be identified, thereby obtaining a first feature representation of the local data.

[0016] The first output feature of the encoder network and the first feature representation of the local data are concatenated to obtain the first input feature of the decoder network.

[0017] The first input features are extracted using the decoder network to obtain the target features of the data to be identified.

[0018] In an exemplary embodiment of this disclosure, before performing fusion feature extraction on the input data using the preset neural network, the method further includes:

[0019] The encoder network and the local feature extraction network in the preset neural network are trained; wherein the initial parameter values ​​of the encoder network are the same as the initial parameter values ​​of the local feature extraction network.

[0020] In one exemplary embodiment of this disclosure, training the encoder network and the local feature extraction network in the preset neural network includes:

[0021] The parameters of the encoder network and the local feature extraction network are iterated according to the preset parameter mapping relationship. When the iteration termination condition is met, the training of the encoder network and the local feature extraction network is completed.

[0022] The parameter mapping relationship is used to determine the parameters of the iterated encoder network and the parameters of the corresponding local feature extraction network.

[0023] In one exemplary embodiment of this disclosure, obtaining input data for a preset neural network based on the data to be identified and local data within the data to be identified includes:

[0024] Based on the data to be identified, a matrix representation of the local data in the data to be identified is constructed to obtain the local data matrix;

[0025] The data to be identified and the corresponding elements in the local data matrix are combined to obtain combined data;

[0026] The combined data is used as input data for the encoder network.

[0027] In one exemplary embodiment of this disclosure, combining the data to be identified and the corresponding elements in the local data matrix to obtain combined data includes:

[0028] according to:

[0029]

[0030] The corresponding elements in the data to be identified X and the local data matrix Y are combined to obtain the combined data Z; where Z(i,j,k) represents the element value at the spatial position corresponding to the i-th row, j-th column, and k-th channel in the combined data Z. Represent the image data X and the local data matrix Y, respectively. line, number The element value at the spatial location corresponding to the column and the k-th channel. This indicates the rounding up operation. These are the combination coefficients.

[0031] In one exemplary embodiment of this disclosure, the step of using the preset neural network to perform fusion feature extraction on the input data to obtain the target features of the data to be identified includes:

[0032] The encoder network is used to extract features from the combined data to obtain the second output feature of the encoder network. The first downsampling layer of the encoder network includes multiple dilated convolutional layers and a pooling layer.

[0033] The second output features of the encoder network are convolved to obtain the second input features of the decoder network;

[0034] The second input feature is extracted using the decoder network to obtain the target feature of the data to be identified.

[0035] In one exemplary embodiment of this disclosure, the preset neural network further includes a local feature fusion network, which includes multiple dilated convolutional layers and a pooling layer; the step of using the preset neural network to extract fused features from the input data to obtain the target features of the data to be identified includes:

[0036] The combined data is input into the local feature fusion network to obtain fused data;

[0037] The encoder and decoder networks in the preset neural network are used to extract features from the fused data to obtain the target features of the data to be identified.

[0038] In one exemplary embodiment of this disclosure, the preset neural network further includes a classifier; after obtaining the target features of the data to be identified, the method further includes:

[0039] The classifier is used to classify and predict the target features of the data to be identified, thereby obtaining the classification result of the data to be identified.

[0040] According to a second aspect of this disclosure, a feature extraction apparatus is provided, comprising:

[0041] The input data generation module is used to obtain the input data of a preset neural network based on the data to be identified and local data in the data to be identified;

[0042] The target feature extraction module is used to extract fused features from the input data using the preset neural network to obtain the target features of the data to be identified.

[0043] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.

[0044] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method described in any of the preceding methods by executing the executable instructions.

[0045] The exemplary embodiments disclosed herein may have some or all of the following beneficial effects:

[0046] In the feature extraction method provided in the exemplary embodiments of this disclosure, input data for a preset neural network is obtained based on the data to be identified and local data within the data to be identified; the preset neural network is then used to perform fusion feature extraction on the input data to obtain the target features of the data to be identified. This disclosure, by introducing local data into the neural network and using the neural network to perform fusion feature extraction on global and local data, can obtain enhanced global features, improving the feature accuracy of neural network recognition and thus enhancing the overall accuracy of neural network recognition.

[0047] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0049] Figure 1 A schematic diagram of an exemplary system architecture for a feature extraction method and apparatus to which embodiments of the present disclosure can be applied is shown;

[0050] Figure 2 A schematic diagram illustrating image segmentation according to an embodiment of the present disclosure is shown.

[0051] Figure 3 A flowchart illustrating a feature extraction method according to an embodiment of the present disclosure is shown schematically;

[0052] Figure 4 A schematic diagram illustrating an improved network structure of a U-Net network according to an embodiment of the present disclosure is shown.

[0053] Figure 5 A schematic diagram illustrating an improved network structure of a U-Net network according to another embodiment of the present disclosure is shown.

[0054] Figure 6 A schematic diagram illustrating the structure of a perforated convolutional layer according to an embodiment of the present disclosure is shown.

[0055] Figure 7 The illustration shows a schematic diagram of a convolution operation between matrices according to an embodiment of the present disclosure;

[0056] Figure 8 A flowchart illustrating fusion feature extraction according to an embodiment of the present disclosure is shown schematically;

[0057] Figure 9 A flowchart illustrating fusion feature extraction according to another embodiment of the present disclosure is shown schematically;

[0058] Figure 10 The diagram illustrates an improved network structure of a U-Net network according to yet another embodiment of the present disclosure.

[0059] Figure 11 A flowchart illustrating the extraction of fusion features according to yet another embodiment of the present disclosure is shown.

[0060] Figure 12A block diagram of a feature extraction apparatus according to an embodiment of the present disclosure is shown schematically;

[0061] Figure 13 The schematic diagram illustrates a structural schematic of a computer system suitable for implementing an electronic device according to embodiments of the present disclosure. Detailed Implementation

[0062] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0063] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0064] Figure 1 A schematic diagram of a system architecture for an exemplary application environment in which a feature extraction method and apparatus according to embodiments of the present disclosure can be applied is shown.

[0065] like Figure 1 As shown, system architecture 100 may include one or more of terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables. Terminal devices 101, 102, and 103 may be various electronic devices, including but not limited to desktop computers, laptops, smartphones, and tablets. It should be understood that... Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. For example, server 105 can be a single server, a server cluster consisting of multiple servers, a cloud computing platform, or a virtualization center. Specifically, server 105 can be used to perform: obtaining input data for a preset neural network based on the data to be identified and local data within the data to be identified; and using the preset neural network to perform fusion feature extraction on the input data to obtain the target features of the data to be identified.

[0066] The feature extraction method provided in this embodiment is generally executed by server 105, and correspondingly, the feature extraction device is generally disposed on server 105. For example, server 105 can send the target features of the data to be identified output by a preset neural network model to a terminal device, and display them to the user through the terminal device. However, it is readily understood by those skilled in the art that the feature extraction method provided in this embodiment can also be executed by one or more of terminal devices 101, 102, and 103, and correspondingly, the feature extraction device can also be disposed in terminal devices 101, 102, and 103. For example, after the feature extraction method is executed by terminal device 101, the extracted target features of the data to be identified can be directly displayed on the display screen of the terminal device. This exemplary embodiment does not impose any special limitations on this.

[0067] The technical solutions of the embodiments of this disclosure are described in detail below:

[0068] In this exemplary embodiment of the disclosure, an example of image segmentation using a neural network can be used for illustration. For example... Figure 2 As shown, Figure (A) is the image to be identified, specifically the plants and animals within it. Specifically, the image can be set as three-dimensional data and used as input data for a neural network to obtain output data of the same dimension. For example, the input data could be: number of image rows (h) * number of image columns (w) * number of image channels, and the output data could be: number of image rows (h) * number of image columns (w) * number of image classes. Figures (B) and (C) illustrate the classification results under accurate identification, where Figure (B) shows the classification result for leopards, and Figure (C) shows the classification result for trees.

[0069] However, in practical applications, the feature accuracy of extracted image features has certain limitations. For example, neural networks may easily identify the leopard's ears in Figure (A) as background, thus reducing the accuracy of neural network recognition.

[0070] To address one or more of the aforementioned problems, this exemplary embodiment provides a feature extraction method. This method can be applied to the server 105, or to one or more of the terminal devices 101, 102, and 103, without special limitation in this exemplary embodiment. (See reference...) Figure 3 As shown, the feature extraction method may include steps S310 and S320:

[0071] Step S310. Obtain the input data of the preset neural network based on the data to be identified and the local data in the data to be identified;

[0072] Step S320. Use the preset neural network to perform fusion feature extraction on the input data to obtain the target features of the data to be identified.

[0073] In the feature extraction method provided in the exemplary embodiments of this disclosure, input data for a preset neural network is obtained based on the data to be identified and local data within the data to be identified; the preset neural network is then used to perform fusion feature extraction on the input data to obtain the target features of the data to be identified. This disclosure, by introducing local data into the neural network and using the neural network to perform fusion feature extraction on global and local data, can obtain enhanced global features, improving the feature accuracy of neural network recognition and thus enhancing the overall accuracy of neural network recognition.

[0074] The steps described above in this example implementation will now be explained in more detail.

[0075] In step S310, the input data of the preset neural network is obtained based on the data to be identified and the local data in the data to be identified.

[0076] Taking image segmentation as an example, the corresponding data to be identified is image data, and the local data in the data to be identified is a portion of the image data containing features with high distinguishability. In the example implementation of this disclosure, the classification result corresponding to the local data in the data to be identified can be determined based on prior knowledge. Here, prior knowledge can refer to the special properties of the research object. For example, prior knowledge can be used to determine... Figure 2 (A) The portion of the image data with larger pixel values ​​belongs to a specific category, such as the tree category.

[0077] For example, when inputting the data to be identified into a preset neural network for feature extraction, the data can be represented by a three-dimensional matrix, such as h*w*channels, to indicate that the data is image data. Similarly, when using local data from the data to be identified as input data to the preset neural network, the local data can be initialized as a three-dimensional matrix of all zeros, denoted as the local data matrix, with a size equal to the output target feature size, h*w*classes. If the classification of some pixels in the data to be identified can be determined based on prior knowledge, the elements of the classification channels corresponding to those pixels can be assigned values ​​in the local data matrix. For example, the element can be assigned the probability value that the pixel is identified as a positive class, and the probability value can be in the range [0, 1]. For example, if the correct classification of the pixel is a leopard, the corresponding element in the local data matrix can be assigned the probability value that the pixel is identified as a leopard, such as a probability value of 0.8.

[0078] The preset neural network may include at least an encoder network and a decoder network. The encoder network may include multiple downsampling layers, each of which includes at least multiple convolutional layers and one pooling layer. The decoder network may include multiple upsampling layers, each of which includes at least one deconvolutional layer and multiple convolutional layers. It is understood that, depending on implementation needs, the number of downsampling and upsampling layers, as well as the number of convolutional layers included in the downsampling layers, can be arbitrary, and this disclosure does not impose specific limitations in this regard. For example, the preset neural network may be a U-Net network, a Transformer network, or other network models that include an encoder network and a decoder network; this disclosure does not impose specific limitations in this regard.

[0079] In one example implementation, the preset neural network may further include a local feature extraction network. Specifically, the local feature extraction network can be constructed based on the encoder network, wherein the network structure of the local feature extraction network is the same as that of the encoder network, and the data to be identified is used as the input data of the encoder network, while local data of the data to be identified is used as the input data of the local feature extraction network.

[0080] This example can be illustrated using the improved U-Net network as the preset neural network. (Reference) Figure 4The diagram illustrates a schematic representation of the improved U-Net network structure. This U-Net network includes a decoder network 202 and a two-stream encoder, namely encoder network 201 and encoder network 203. Encoder network 203 is a local feature extraction network constructed based on encoder network 201. It can be seen that encoder network 201 and encoder network 203 have identical network structures. Encoder network 201 includes three downsampling layers (2011, 2012, and 2013), and decoder network 202 includes three upsampling layers (2022, 2023, and 2024). It should be noted that this disclosure does not specifically limit the number of downsampling and upsampling layers, but the number of downsampling and upsampling layers is the same. Three feature propagation layers (2031, 2032, and 2033) are established between encoder network 201 and decoder network 202. For encoder network 201, features are extracted from the data to be recognized through the three downsampling layers, and the extracted feature maps are transmitted to decoder network 202 by the three feature propagation layers. For the decoder network 202, feature extraction of the data to be identified can be performed through three upsampling layers, and feature fusion of the data to be identified can also be performed.

[0081] Specifically, regarding the network structure of each downsampling layer in the encoder network 201, taking downsampling layer 2011 as an example, downsampling layer 2011 includes two convolutional layers and one pooling layer. The convolutional layers can be shallow convolutional layers, which have smaller receptive fields and smaller overlapping areas, ensuring that the neural network captures more details. The convolutional layers have learnable parameters, which can be fitted by training a pre-defined neural network. The pooling layer can be a max pooling layer or an average pooling layer. In the encoder network 201, horizontal hollow arrows indicate convolution processing performed by convolutional layers, and downward solid arrows indicate max pooling downsampling processing performed by max pooling layers, or average pooling downsampling processing performed by average pooling layers. Regarding the network structure of each upsampling layer in the decoder network 202, taking upsampling layer 2022 as an example, upsampling layer 2022 includes one deconvolutional layer and two convolutional layers. Both the convolutional and deconvolutional layers have learnable parameters, which can also be fitted by training a pre-defined neural network. In the decoder network 202, horizontal hollow arrows indicate convolution processing performed by convolutional layers, and upward solid arrows indicate deconvolution upsampling processing performed by deconvolutional layers.

[0082] The data to be identified can be used as input data to encoder network 201 to obtain global features of the data. Local data of the data to be identified can be used as input data to local feature extraction network (encoder network 203) to obtain local features of the data. These local features are then propagated to encoder network 201 to concatenate the channel dimensions of the global and local features of the data to be identified, resulting in... Figure 4 The feature map 2014 shown is used as an example. A convolution operation is performed on the concatenated feature map 2014 to obtain the input to the decoder network 202. Figure 4 The feature map 2021 is shown in the figure. In other examples, the global and local features of the data to be identified can also be propagated to the feature map parameters by summing element by element, that is, the local features of the data to be identified are transmitted to the encoder network 201. This disclosure does not limit this.

[0083] Figure 4 The network structure shown possesses scalability and fault tolerance. This structure can be extended to all neural networks containing encoder-decoder structures, enabling feature extraction from both the data to be identified and its local data. Simultaneously, by using the local data of the data to be identified as input to the neural network, the output feature map can more accurately describe the data to be identified, thereby improving the feature accuracy of the neural network. The newly added network branch, encoder network 203, simultaneously extracts features from the local data of the data to be identified and integrates the extracted local features into encoder network 201. This allows decoder network 202 to fuse the global and local features of the data to be identified, improving the operating efficiency of the neural network.

[0084] In another example implementation, the first downsampling layer of the encoder network in the preset neural network may include multiple dilated convolutional layers and a pooling layer. In this example, the preset neural network is still described using the improved U-Net network. (See reference...) Figure 5 The diagram illustrates another improved U-Net network structure, which includes an encoder network 201 and a decoder network 202. It should be noted that... Figure 5 The downsampling layer 2011 of the encoder network 201 shown consists of two dilated convolutional layers (2041 and 2042) and a pooling layer (2043). Figure 6The diagram illustrates the structure of a dilated convolutional layer. By injecting holes into the kernel of a standard convolution, the dilated convolutional layer increases the receptive field of the convolution operation, allowing the output to contain a wider range of information. In this example, the hyperparameters of the dilated convolutional layer can be initialized, such as setting the dilation rate r = 2 and the stride s = 2. The dilation rate defines the distance between elements of the convolution kernel. It is understood that this disclosure does not limit the specific values ​​of the dilation rate and stride of the dilated convolutional layer. Furthermore, the pooling layer 2043 in the downsampling layer 2011 can be either an average pooling layer or a max pooling layer; this disclosure also does not limit its use.

[0085] In this example, a matrix representation of local data within the data to be identified can be constructed based on the data to be identified, resulting in a local data matrix. The corresponding elements of the data to be identified and the local data matrix are then combined to obtain combined data, which is used as... Figure 5 The input data of the encoder network 201 shown. Specifically, refer to... Figure 7 As shown, when the data to be identified is image data, it can be represented as a three-dimensional matrix to obtain image data X, which is h*w*channels. Simultaneously, prior knowledge can be digitized based on the size of the image data matrix, that is, local data in the data to be identified can be represented as a three-dimensional matrix to obtain local data matrix Y, which is also h*w*channels. Both matrices have the same size.

[0086] The corresponding elements in the image data X and the local data matrix Y can be combined according to formula (1). Specifically, it can be done according to:

[0087]

[0088] The combined data Z is obtained, and the matrix size of combined data Z is (2h)*(2w)*channels. Here, Z(i,j,k) represents the element value at the spatial position corresponding to the i-th row, j-th column, and k-th channel in combined data Z. Representing the image data X and the local data matrix Y respectively, the first... line, number The element value at the spatial location corresponding to the column and the k-th channel. This indicates the rounding up operation. These are the combination coefficients.

[0089] For example, for the element values ​​at the spatial positions corresponding to the 1st row, 1st column, and kth channel in the combined data Z, i.e., i=1, j=1, the combination coefficients... At this point, the element values ​​at the spatial positions corresponding to the 1st row, 1st column, and kth channel in image data X can be assigned the element values ​​at the spatial positions corresponding to the 1st row, 1st column, and kth channel in combined data Z. Similarly, for the element values ​​at the spatial positions corresponding to the 1st row, 2nd column, and kth channel in combined data Z, i.e., when i=1 and j=2, the combination coefficients... At this point, the element values ​​at the spatial positions corresponding to the 1st row, 2nd column, and kth channel in the local data matrix Y can be assigned the element values ​​at the spatial positions corresponding to the 1st row, 2nd column, and kth channel in the combined data Z. The resulting combined data Z is as follows: Figure 7 As shown.

[0090] Figure 5 The network structure shown does not add new network branches. Instead, it replaces the shallow convolutional layers in the original encoder network with dilated convolutional layers, and combines them with pooling layers to achieve parallel fusion between the data to be identified and its local data. Compared to Figure 4 The network structure shown can not only improve the training and running efficiency of neural networks, but also keep the neural network lightweight.

[0091] In step S320, the preset neural network is used to extract fusion features from the input data to obtain the target features of the data to be identified.

[0092] The default neural network is Figure 4 When using the U-Net network structure shown, the data to be identified can be used as input data to encoder network 201, and local data of the data to be identified can be used as input data to encoder network 203. Encoder network 201 extracts features from the data to be identified to obtain global features, and encoder network 203 extracts features from the local data to obtain local features. Then, decoder network 202 fuses the global and local features of the data to be identified to obtain the target features.

[0093] In one example implementation, reference Figure 8 As shown, steps S810 to S840 can be performed via... Figure 4 The network structure of the U-Net network shown performs fusion feature extraction on the input data to obtain the target features of the data to be identified.

[0094] Step S810. Use the encoder network to extract features from the data to be identified to obtain the first output feature of the encoder network.

[0095] For example, when the data to be identified is image data, the image data can be input into the downsampling layer 2011 of the encoder network 201. The input image data undergoes two convolutional processes through the two convolutional layers in the downsampling layer 2011 to obtain a first feature map. This first feature map is then transmitted to the upsampling layer 2024 in the decoder network 202 through the feature propagation layer 2031. Simultaneously, in the encoder network 201, the first feature map can be downsampled through the pooling layer in the downsampling layer 2011; for example, max pooling downsampling can be performed on the first feature map. Then, the downsampled first feature map is input into the downsampling layer 2012. It is understood that the processing of the input feature map by the downsampling layers 2012 and 2013 is the same as the processing of the image data by the downsampling layer 2011, and will not be described in detail here. Similarly, a second feature map can be obtained through the downsampling layer 2012, and this second feature map is transmitted to the upsampling layer 2023 in the decoder network 202 through the feature propagation layer 2032. Finally, the third feature map can be obtained through the downsampling layer 2013, and then transmitted to the upsampling layer 2022 in the decoder network 202 through the feature propagation layer 2033. The third feature map is the first output feature of the encoder network 201.

[0096] As the encoder network deepens, the feature extraction of image data gradually shifts from local description to global description, which can more accurately describe the image data and thus help ensure the accuracy of image segmentation.

[0097] Step S820. Use the local feature extraction network to extract features from the local data in the data to be identified, and obtain the first feature representation of the local data.

[0098] Corresponding to step S810, the local data in the data to be identified is a portion of the image data. For example, this portion of data can be represented by a local data matrix. The local feature extraction network is encoder network 203, whose network structure is the same as encoder network 201. The encoder network 203 can be used to extract features from the local data matrix to obtain the first feature representation of the local data matrix. The specific feature extraction process can be referred to step S810, and will not be described in detail here.

[0099] Step S830. The first output feature of the encoder network and the first feature representation of the local data are concatenated to obtain the first input feature of the decoder network.

[0100] The first feature representation of the local data can be concatenated with the first output feature of the encoder network. For example, the channel dimensions of the two can be concatenated, or they can be concatenated by element-wise summation to achieve feature propagation. Concatenating the first feature representation of the local data with the first output feature of the encoder network yields, for example... Figure 4 After obtaining the feature map 2014 shown, a convolution operation can be performed on the feature map 2014 to obtain the following result: Figure 4 The feature map 2021 shown is used as the first input feature of the decoder network 202, so as to perform feature fusion on the first input feature using the decoder network 202.

[0101] Step S840. Use the decoder network to extract features from the first input features to obtain the target features of the data to be identified.

[0102] First, feature map 2021 can be input as the first feature map to be fused into the upsampling layer 2022 of the decoder network 202. The deconvolution layer in upsampling layer 2022 performs deconvolution upsampling on feature map 2021, which means fusing the first output feature of the encoder network and the first feature representation of the local data contained in feature map 2021 to obtain the first fused feature map. The first fused feature map is then concatenated with the third feature map transmitted through the feature propagation layer. The concatenated feature map is then subjected to two convolutional processes by two convolutional layers in upsampling layer 2022 to obtain the second feature map to be fused. The second feature map to be fused is then input into upsampling layer 2023, where deconvolutional upsampling is performed. It can be understood that the processing of the input feature map by upsampling layers 2023 and 2024 is the same as the processing of feature map 2021 by upsampling layer 2022, and will not be elaborated further here. Similarly, the third feature map to be fused can be obtained through the upsampling layer 2023. The third feature map to be fused is then input into the upsampling layer 2024, and deconvolution upsampling and two convolution processes are performed in sequence. The output data of the upsampling layer 2024 is the target feature of the image data.

[0103] In this example, local data of the data to be identified is used as part of the input data to a preset neural network containing a two-stream encoder. The two-stream encoder extracts both global and local features of the data to be identified. These global and local features are then fused through the decoder network in the preset neural network, resulting in feature enhancement of the global features and improving the feature accuracy of the preset neural network. Furthermore, when using a preset neural network containing a two-stream encoder for recognition applications in related fields, such as image recognition and face recognition, the accuracy of the preset neural network can be improved.

[0104] pass Figure 4 The U-Net network structure shown can be trained before fusing and extracting features from the input data. For example, a dataset consisting of input and output data can be used to fit the parameters. After parameter fitting, feature extraction, image segmentation, and other tasks can be performed using this network structure.

[0105] In one example implementation, such as Figure 4 As shown, the U-Net network comprises three parts: an encoder network (201), a local feature extraction network (i.e., encoder network 203), and a decoder network (202). This example mainly describes the training process of the encoder network and the local feature extraction network. It is understandable that while training the encoder network and the local feature extraction network, the parameters of the decoder network are also continuously iterated to fit and obtain better parameters. For example, during the training process of the encoder network and the local feature extraction network, a parameter interconnection mechanism can be added to the training process of this part of the network structure, that is, establishing a mapping relationship between the parameters of the two-stream encoder to improve the training efficiency of this network structure. The initial values ​​of the encoder network parameters are the same as the initial values ​​of the local feature extraction network parameters.

[0106] Specifically, the encoder network has learnable convolutional layer parameters. The encoder network 201 and the local feature extraction network 203 have the same network structure; therefore, the parameter sizes of their corresponding convolutional and pooling layers are also the same. Correspondingly, the parameter set of the encoder network 201 is surjective to the parameter set of the local feature extraction network 203, meaning that each parameter in the local feature extraction network 203 has a corresponding parameter in the encoder network 201. For example, let X represent the parameter set of the encoder network 201 and Y represent the parameter set of the local feature extraction network 203. The parameter mapping relationship between the encoder network 201 and the local feature extraction network 203 is expressed as:

[0107] Y(i)=f[X(i)] (2)

[0108] The parameter mapping relationship can be used to determine the parameters of the iterative encoder network and the corresponding parameters of the local feature extraction network.

[0109] For example, the parameter mapping relationship between the encoder network 201 and the local feature extraction network 203 can be preset as follows:

[0110]

[0111] Where X1(i) represents the i-th parameter of the encoder network 201 after the current iteration, Y1(i) represents the i-th parameter of the local feature extraction network 203 after the current iteration, X2(i) represents the i-th parameter of the encoder network 201 used in the next iteration, and Y1(i) represents the i-th parameter of the local feature extraction network 203 used in the next iteration. It should be noted that the initial values ​​of the encoder network parameters are the same as those of the local feature extraction network parameters. After gradient updates of the parameters of the local feature extraction network 203 and the encoder network 201, the parameters in both networks will change. In this example, a parameter mapping relationship can be used to ensure that the parameters of the encoder network used in each iteration are the same as the parameters of the corresponding local feature extraction network. Specifically, the average value of X1(i) and Y1(i) can be used as the i-th parameter of the encoder network 201 and the local feature extraction network 203 in the next iteration.

[0112] It should be noted that during training, the initial parameter values ​​of the encoder network and the local feature extraction network should be the same. Using the same initial parameter values ​​for both networks will cause the parameters to change continuously during training. However, by adjusting the parameters in the encoder and local feature extraction networks using a parameter mapping relationship, a definite mapping relationship can be maintained between them during parameter fitting, facilitating the acquisition of optimal parameters.

[0113] For example, the parameters of the encoder network and the local feature extraction network can be iterated according to a preset parameter mapping relationship. When the iteration termination condition is met, the training of the encoder network and the local feature extraction network is completed. For instance, an objective function can be constructed based on the input and output data. Based on this objective function, the parameters of the encoder network and the local feature extraction network can be iteratively updated using the stochastic gradient descent algorithm. When the iteration termination condition is met, the training of the encoder network and the local feature extraction network is completed. The iteration termination condition can be that the objective function converges, completing the training of all parameters, or it can be achieved by iteratively updating the parameters in reverse order, completing the training of all parameters when a preset number of iterations is met. This disclosure does not limit this approach.

[0114] In this example, there is no feature propagation layer between the layers of the encoder network and the local feature extraction network; the output features of the encoder network and the local feature extraction network are simply concatenated. Therefore, during forward inference, the inference results of each layer of the encoder network and the local feature extraction network are relatively independent. Furthermore, if the encoder network and the local feature extraction network have different initial parameter values, the features they extract will also be different. Additionally, if there is no mapping relationship between the parameters of the encoder network and the parameters of the local feature extraction network, information cannot be transferred between them. Based on this, when performing parameter fitting, the gradient update of the parameters is only affected by the network structure, resulting in poor parameter fitting performance. It can be seen that the parameter interconnection mechanism can achieve feature sharing between different network structures, facilitating the fitting of better parameters. Furthermore, in the parameter interconnection mechanism, the parameters of the encoder network and the parameters of the local feature extraction network have a definite parameter mapping relationship. For example, the parameters of the encoder network can be calculated from the parameters of the local feature extraction network. When storing parameters, only the parameters of the encoder network or the parameters of the local feature extraction network need to be stored, saving storage space and making the neural network structure more lightweight.

[0115] The default neural network is Figure 5 When constructing the network structure of the U-Net network as shown, a matrix representation of the local data in the data to be identified can be built based on the data to be identified, resulting in a local data matrix. The corresponding elements of the data to be identified and the local data matrix are then combined to obtain combined data, which is then used as... Figure 5 The input data of the encoder network 201 shown is used. The encoder network 201 and decoder network 202 are used to perform fusion feature extraction on the combined data to obtain the target features of the data to be identified.

[0116] In another example implementation, refer to Figure 9 As shown, steps S910 to S930 can be performed via... Figure 5 The network structure of the U-Net network shown performs fusion feature extraction on the input data to obtain the target features of the data to be identified.

[0117] Step S910. Use the encoder network to extract features from the combined data to obtain the second output features of the encoder network. The first downsampling layer of the encoder network includes multiple dilated convolutional layers and a pooling layer.

[0118] For example, the data to be identified is image data, and the local data in the data to be identified is a portion of the image data. When represented by a local data matrix, combined data is obtained from the image data and the local data matrix. This combined data can be input into the first downsampling layer 2011 of the encoder network 201. The downsampling layer 2011 consists of two dilated convolutional layers (2041 and 2042) and a pooling layer (2043). The input image data is convolved twice by the two dilated convolutional layers in the downsampling layer 2011 to obtain a fourth feature map. This fourth feature map is then transmitted to the upsampling layer 2024 in the decoder network 202 through the feature propagation layer 2031. Simultaneously, in the encoder network 201, the fourth feature map can be downsampled by the pooling layer in the downsampling layer 2011; for example, average pooling downsampling can be performed on the fourth feature map. It should be noted that after performing convolution operations on the combined data using dilated convolutional layers, the image data part and the local data matrix part in the combined data still maintain a relatively independent spatial distribution state until the pooling layer merges the image data and the local data matrix.

[0119] Then, the downsampled fourth feature map is input into downsampling layer 2012. It's understood that downsampling layers 2012 and 2013 process the input feature map in the same way downsampling layer 2011 processes the image data; this will not be elaborated upon here. Similarly, the fifth feature map is obtained through downsampling layer 2012 and transmitted to upsampling layer 2023 in decoder network 202 via feature propagation layer 2032. Finally, the sixth feature map is obtained through downsampling layer 2013, and the third feature map is transmitted to upsampling layer 2022 in decoder network 202 via feature propagation layer 2033. The sixth feature map is the second output feature of encoder network 201.

[0120] Step S920. Perform convolution operation on the second output features of the encoder network to obtain the second input features of the decoder network.

[0121] Figure 5 Feature map 2015 shown is the second output feature of encoder network 201. Convolution operations can be performed on feature map 2015 to obtain... Figure 5 The feature map 2020 is shown, and the feature map 2020 is used as the second input feature of the decoder network 202 to perform feature fusion on the second input feature using the decoder network 202.

[0122] Step S930. Use the decoder network to extract features from the second input features to obtain the target features of the data to be identified.

[0123] Feature map 2020 can be input as the fourth feature map to be fused into the upsampling layer 2022 of the decoder network 202. The deconvolution layer in upsampling layer 2022 performs deconvolution upsampling on feature map 2020, which means fusing the image data and local data matrix contained in feature map 2020 to obtain the fourth fused feature map. The fourth fused feature map is then concatenated with the third feature map transmitted through the feature propagation layer. The concatenated feature map is then subjected to two convolutional processes by two convolutional layers in upsampling layer 2022 to obtain the fifth feature map to be fused. The fifth feature map to be fused is then input into upsampling layer 2023, where deconvolutional upsampling is performed. It can be understood that the processing of the input feature map by upsampling layers 2023 and 2024 is the same as the processing of feature map 2020 by upsampling layer 2022, and will not be elaborated further here. Similarly, the sixth feature map to be fused can be obtained through the upsampling layer 2023. The sixth feature map to be fused is then input into the upsampling layer 2024, and deconvolution upsampling and two convolution processes are performed in sequence. The output data of the upsampling layer 2024 is the target feature of the image data.

[0124] In this example, the data to be identified and its local data are input together into the encoder network of the preset neural network. The encoder network contains a downsampling layer with two dilated convolutional layers and a pooling layer, which can fuse the data to be identified and its local data. The decoder network can also fuse the data to be identified and its local data, thereby achieving parallel fusion of the local data of the data to be identified, further enhancing the global features and improving the feature accuracy of the preset neural network.

[0125] Due to reasons such as model reuse or structural conflicts, it is impossible to modify the network structure of neural networks as follows: Figure 4 and Figure 5 The improvements shown are as follows. (See reference.) Figure 10 The diagram illustrates another improved preset neural network structure, which may include an encoder network and a decoder network, and may also include a local feature fusion network, which includes multiple dilated convolutional layers and a pooling layer. For example, the local feature fusion network may include, as shown below... Figure 5The encoder network 201 shown has a first downsampling layer 2011, which consists of two dilated convolutional layers (2041 and 2042) and a pooling layer (2043). The pooling layer 2043 can be an average pooling layer or a max pooling layer, and this disclosure does not limit it in this way. Furthermore, a local feature fusion network can be connected in series with the encoder network and the decoder network to fuse local data of the data to be recognized into a preset neural network.

[0126] Based on the data to be identified, a matrix representation of the local data within the data to be identified can be constructed to obtain a local data matrix. The corresponding elements of the data to be identified and the local data matrix can then be combined to obtain combined data. This combined data can be used as input to the local feature fusion network, specifically the downsampling layer 2011. The pooling layer 2043 within the downsampling layer 2011 fuses the data to be identified and the local data within the combined data to obtain fused data. Finally, the fused data undergoes feature extraction and further feature fusion through the encoder and decoder networks to obtain the output data.

[0127] In another example implementation, refer to Figure 11 As shown, steps S1110 to S1120 can be performed via... Figure 10 The network structure of the preset neural network shown performs fusion feature extraction on the input data to obtain the target features of the data to be identified.

[0128] Step S1110. Input the combined data into the local feature fusion network to obtain fused data.

[0129] For example, the data to be identified is image data, and the local data in the data to be identified is a portion of the image data. When represented by a local data matrix, combined data is obtained from the image data and the local data matrix. In this example, the local feature fusion network is... Figure 10 The downsampling layer 2011 shown can be used to input combined data. The input image data undergoes two convolution processes through two dilated convolutional layers in downsampling layer 2011 to obtain the fourth feature map. The fourth feature map can then be downsampled using pooling layers in downsampling layer 2011; for example, average pooling downsampling can be performed on the fourth feature map to obtain fused data.

[0130] Step S1120. Use the encoder network and decoder network in the preset neural network to perform feature fusion on the fused data to obtain the target features of the data to be identified.

[0131] The fused data is used as input to the encoder network. The encoder and decoder networks then perform feature extraction and further feature fusion on this fused data, ultimately obtaining the target features of the image data. The encoder and decoder networks are used as... Figure 4 Taking the encoder network 201 and decoder network 202 shown as examples, after using the fused data as the input data of the encoder network 201, the feature extraction process and feature fusion process of the encoder network 201 and decoder network 202 on the fused data can be referred to steps 810 and 840, which will not be repeated here.

[0132] In this example, the local feature fusion network serves as the input layer of the pre-defined neural network, while the original network structure, consisting of the encoder and decoder networks, becomes the intermediate and output layers of the pre-defined neural network. Through the improved neural network, the data to be identified and its local data are further fused. This provides a new and improved method for integrating the local data of the data to be identified into the encoder-decoder neural network, while maintaining a relatively constant computational load.

[0133] In one example implementation, the preset neural network may further include a classifier, which can be placed after the decoder network. For example, the classifier could be a Softmax classifier, a sigmoid classifier, etc. The classifier can be used to classify and predict the target features of the data to be identified, thus obtaining the classification result of the data. For instance, when the data to be identified is image data, after the decoder network outputs the target features of the image data, the classifier can be used to calculate the probability that pixels in the target features belong to different categories, thereby predicting the category of the pixels in the target features and performing pixel classification, i.e., image segmentation.

[0134] It should be noted that the feature extraction method disclosed herein can be applied to various scenarios with feature extraction requirements. This disclosure only illustrates feature extraction in the scenario of image segmentation.

[0135] In the feature extraction method provided in the exemplary embodiments of this disclosure, input data for a preset neural network is obtained based on the data to be identified and local data within the data to be identified; the preset neural network is then used to perform fusion feature extraction on the input data to obtain the target features of the data to be identified. This disclosure, by introducing local data into the neural network and using the neural network to perform fusion feature extraction on global and local data, can obtain enhanced global features, improving the feature accuracy of neural network recognition and thus enhancing the overall accuracy of neural network recognition.

[0136] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0137] Furthermore, this example embodiment also provides a feature extraction device. This device can be applied to a terminal device or a server. (See reference...) Figure 12 As shown, the feature extraction device 1200 may include an input data generation module 1210 and a target feature extraction module 1220, wherein:

[0138] The input data generation module 1210 is used to obtain input data for a preset neural network based on the data to be identified and local data in the data to be identified;

[0139] The target feature extraction module 1220 is used to extract fusion features from the input data using the preset neural network to obtain the target features of the data to be identified.

[0140] In one optional implementation, the preset neural network in the feature extraction device 1200 includes at least an encoder network and a decoder network. The encoder network includes multiple downsampling layers, each of which includes at least multiple convolutional layers and a pooling layer. The decoder network includes multiple upsampling layers, each of which includes at least a deconvolutional layer and multiple convolutional layers.

[0141] In an optional embodiment, the preset neural network further includes a local feature extraction network, the network structure of which is the same as that of the encoder network; the input data generation module 1210 includes:

[0142] The extraction network construction submodule is used to construct the local feature extraction network based on the encoder network, wherein the network structure of the local feature extraction network is the same as the network structure of the encoder network.

[0143] The first input data generation submodule is used to use the data to be identified as input data of the encoder network; and to use local data of the data to be identified as input data of the local feature extraction network.

[0144] In one optional implementation, the target feature extraction module 1220 includes:

[0145] The first feature extraction submodule is used to extract features from the data to be identified using the encoder network to obtain the first output feature of the encoder network;

[0146] The second feature extraction submodule is used to extract features from local data in the data to be identified using the local feature extraction network, so as to obtain a first feature representation of the local data.

[0147] The first input feature generation submodule is used to concatenate the first output feature of the encoder network and the first feature representation of the local data to obtain the first input feature of the decoder network.

[0148] The first target feature generation submodule is used to extract features from the first input features using the decoder network to obtain the target features of the data to be identified.

[0149] In an optional embodiment, the feature extraction device 1200 further includes:

[0150] A network training module is used to train the encoder network and the local feature extraction network in the preset neural network; wherein the initial parameter values ​​of the encoder network are the same as the initial parameter values ​​of the local feature extraction network.

[0151] In one optional implementation, the network training module is configured to iterate the parameters of the encoder network and the parameters of the local feature extraction network according to a preset parameter mapping relationship, and complete the training of the encoder network and the local feature extraction network when the iteration termination condition is met; wherein, the parameter mapping relationship is used to determine the parameters of the iterated encoder network and the corresponding parameters of the local feature extraction network.

[0152] In one optional implementation, the input data generation module 1210 includes:

[0153] The combined data generation submodule is used to construct a matrix representation of the local data in the data to be identified based on the data to be identified, thereby obtaining a local data matrix; and to combine the corresponding elements of the data to be identified and the local data matrix to obtain combined data.

[0154] The second input data generation submodule is used to use the combined data as input data for the encoder network.

[0155] In one alternative implementation, the combined data generation submodule is configured to be used to:

[0156]

[0157] The corresponding elements in the data to be identified X and the local data matrix Y are combined to obtain the combined data Z; where Z(i,j,k) represents the element value at the spatial position corresponding to the i-th row, j-th column, and k-th channel in the combined data Z. Represent the image data X and the local data matrix Y, respectively. line, number The element value at the spatial location corresponding to the column and the k-th channel. This indicates the rounding up operation. These are the combination coefficients.

[0158] In one optional implementation, the target feature extraction module 1220 includes:

[0159] The third feature extraction submodule is used to extract features from the combined data using the encoder network to obtain the second output feature of the encoder network. The first downsampling layer of the encoder network includes multiple dilated convolutional layers and a pooling layer.

[0160] The first input feature generation submodule is used to perform convolution operations on the second output features of the encoder network to obtain the second input features of the decoder network.

[0161] The second target feature generation submodule is used to extract features from the second input features using the decoder network to obtain the target features of the data to be identified.

[0162] In one optional implementation, the preset neural network further includes a local feature fusion network, which includes multiple dilated convolutional layers and a pooling layer; the target feature extraction module 1220 includes:

[0163] A fusion data generation submodule is used to input the combined data into the local feature fusion network to obtain fusion data;

[0164] The third target feature generation submodule is used to perform feature fusion on the fused data using the encoder network and decoder network in the preset neural network to obtain the target features of the data to be identified.

[0165] In one optional implementation, the preset neural network further includes a classifier; the feature extraction device 1200 further includes:

[0166] The data recognition module is used to classify and predict the target features of the data to be recognized through the classifier, and obtain the classification result of the data to be recognized.

[0167] The specific details of each module in the above feature extraction device have been described in detail in the corresponding feature extraction methods, so they will not be repeated here.

[0168] The modules in the above-described device can be general-purpose processors, including central processing units (CPUs), network processors, etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Each module can also be implemented using software, firmware, etc. The processors in the above-described device can be independent processors or integrated together.

[0169] Exemplary embodiments of this disclosure also provide a computer-readable storage medium having a program product stored thereon capable of implementing the methods described above in this specification. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when run on an electronic device, causes the electronic device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) including program code and may run on an electronic device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0170] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0171] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0172] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0173] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0174] Exemplary embodiments of this disclosure also provide an electronic device capable of implementing the above-described method. Referring below... Figure 13 To describe an electronic device 1300 according to such an exemplary embodiment of the present disclosure. Figure 13 The electronic device 1300 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0175] like Figure 13 As shown, the electronic device 1300 can be represented as a general-purpose computing device. The components of the electronic device 1300 may include, but are not limited to: at least one processing unit 1310, at least one storage unit 1320, a bus 1330 connecting different system components (including storage unit 1320 and processing unit 1310), and a display unit 1340.

[0176] Storage unit 1320 stores program code that can be executed by processing unit 1310, causing processing unit 1310 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, processing unit 1310 can perform... Figure 3 , Figure 8 , Figure 9 and Figure 11 Any one or more of the method steps.

[0177] Storage unit 1320 may include readable media in the form of volatile storage units, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0178] Storage unit 1320 may also include a program / utility 1324 having a set (at least one) program module 1325, such program module 1325 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0179] Bus 1330 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0180] Electronic device 1300 can also communicate with one or more external devices 1400 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 1300, and / or with any device that enables electronic device 1300 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1350. Furthermore, electronic device 1300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1360. Figure 13 As shown, network adapter 1360 communicates with other modules of electronic device 1300 via bus 1330. It should be understood that, although... Figure 13 As not shown, other hardware and / or software modules can be used in conjunction with the electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0181] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to the exemplary embodiments of this disclosure.

[0182] Furthermore, the above figures are merely illustrative representations of the processes included in the methods according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0183] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0184] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A feature extraction method, characterized in that, include: The input data of the preset neural network is obtained based on the data to be identified and local data in the data to be identified; The input data is fused and feature extracted using the preset neural network to obtain the target features of the data to be identified. The preset neural network includes at least an encoder network and a decoder network. The preset neural network also includes a local feature fusion network, which includes multiple dilated convolutional layers and a pooling layer. The local feature fusion network is connected in series with the encoder network and the decoder network to fuse local data of the data to be identified into the preset neural network. The step of obtaining input data for a preset neural network based on the data to be identified and local data within the data to be identified includes: Based on the data to be identified, a matrix representation of the local data in the data to be identified is constructed to obtain a local data matrix. In this local data matrix, according to the classification of some pixels in the data to be identified, the elements of the classification channel corresponding to the some pixels are assigned values ​​to determine the local data matrix. The data to be identified and the corresponding elements in the local data matrix are combined to obtain combined data; The combined data is used as the input data for the encoder network; The step of using the preset neural network to perform fusion feature extraction on the input data to obtain the target features of the data to be identified includes: The combined data is input into the local feature fusion network to obtain fused data; The encoder and decoder networks in the preset neural network are used to extract features from the fused data to obtain the target features of the data to be identified.

2. The feature extraction method according to claim 1, characterized in that, The encoder network includes multiple downsampling layers, each of which includes at least multiple convolutional layers and one pooling layer. The decoder network includes multiple upsampling layers, each of which includes at least one deconvolutional layer and multiple convolutional layers.

3. The feature extraction method according to claim 1, characterized in that, The step of combining the data to be identified and the corresponding elements in the local data matrix to obtain combined data includes: according to: The corresponding elements in the data to be identified X and the local data matrix Y are combined to obtain the combined data Z; where, Represents the first element in the combined data Z. i line, number j Column, No. k The element value at the spatial location corresponding to each channel. , Represent the image data X and the local data matrix Y, respectively. line, number Column, No. k The element value at the spatial location corresponding to each channel. This indicates the rounding up operation. , These are the combination coefficients.

4. The feature extraction method according to claim 1, characterized in that, The step of using the preset neural network to perform fusion feature extraction on the input data to obtain the target features of the data to be identified includes: The encoder network is used to extract features from the combined data to obtain the second output feature of the encoder network. The first downsampling layer of the encoder network includes multiple dilated convolutional layers and a pooling layer. The second output features of the encoder network are convolved to obtain the second input features of the decoder network; The second input feature is extracted using the decoder network to obtain the target feature of the data to be identified.

5. The feature extraction method according to claim 1, characterized in that, The preset neural network further includes a classifier; after obtaining the target features of the data to be identified, the method further includes: The classifier is used to classify and predict the target features of the data to be identified, thereby obtaining the classification result of the data to be identified.

6. A feature extraction device, characterized in that, include: An input data generation module is used to obtain input data for a preset neural network based on the data to be identified and local data in the data to be identified. The preset neural network includes at least an encoder network and a decoder network, and also includes a local feature fusion network. The local feature fusion network includes multiple dilated convolutional layers and a pooling layer. The local feature fusion network is connected in series with the encoder network and the decoder network to fuse local data of the data to be identified into the preset neural network. The target feature extraction module is used to perform fusion feature extraction on the input data using the preset neural network to obtain the target features of the data to be identified. The input data generation module is configured to execute: Based on the data to be identified, a matrix representation of the local data in the data to be identified is constructed to obtain a local data matrix. In this local data matrix, according to the classification of some pixels in the data to be identified, the elements of the classification channel corresponding to the some pixels are assigned values ​​to determine the local data matrix. The data to be identified and the corresponding elements in the local data matrix are combined to obtain combined data; The combined data is used as the input data for the encoder network; The target feature extraction module is configured to perform: The combined data is input into the local feature fusion network to obtain fused data; The encoder and decoder networks in the preset neural network are used to extract features from the fused data to obtain the target features of the data to be identified.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-5.

8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-5 by executing the executable instructions.

Images