Target attribute recognition method and related device

Abstract

The present application discloses a target attribute recognition method, a target attribute recognition device, an electronic device and a computer-readable storage medium, the target attribute recognition method comprising: obtaining a component feature set, a component attribute recognition result set and an overall feature of a target to be recognized in an image to be recognized; fusing at least one of the component feature set and the component attribute recognition result set with the overall feature to obtain a first fused feature; and performing overall attribute recognition on the target to be recognized based on the first fused feature to obtain an overall attribute recognition result. The above scheme can improve the accuracy of the overall attribute recognition result.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a target attribute recognition method, a target attribute recognition device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Target attribute recognition is to predict a set of attributes from a predefined attribute list to describe the target. Specifically, target attribute recognition includes obtaining an image to be recognized, extracting target features in the image to be recognized, and performing attribute recognition on the target features to obtain target attribute recognition results.

[0003] However, the attribute recognition results obtained by existing target attribute recognition methods are not accurate. Summary of the invention

[0004] The present application provides a target attribute recognition method, a target attribute recognition device, an electronic device and a computer-readable storage medium, which can solve the problem that the attribute recognition results obtained by the existing target attribute recognition methods are not accurate.

[0005] The present application provides a method for target attribute recognition, comprising: obtaining a component feature set, a component attribute recognition result set and an overall feature of a target to be recognized in an image to be recognized; fusing at least one of the component feature set and the component attribute recognition result set with the overall feature to obtain a first fused feature; and performing overall attribute recognition on the target to be recognized based on the first fused feature to obtain an overall attribute recognition result.

[0006] In the above scheme, the component feature set and the component attribute recognition result set represent the component attributes, the overall feature represents the overall attribute, and the overall attribute is associated with the component attribute. Therefore, compared with the overall attribute recognition based only on the overall feature, in this embodiment, the first fused feature obtained by fusing the component feature set / component attribute recognition result and the overall feature is used for overall attribute recognition, and the correlation between the component attributes and the overall attributes is effectively used in the overall attribute recognition process, so that the component attributes guide the overall attribute recognition process, thereby improving the accuracy of the overall attribute recognition result.

[0007] Further, in some embodiments, at least one of the component feature set and the component attribute recognition result set is fused with the overall feature to obtain a first fused feature, including: fusing the component feature set, the component attribute recognition result set and the overall feature to obtain a first fused feature.

[0008] Compared with fusing only one of the component attribute recognition result set and the component feature set with the overall feature, the above scheme fuses both the component attribute recognition result set and the component feature set with the overall feature, so that the overall attribute recognition process can better capture the relationship between the component attribute and the overall attribute. This further improves the effectiveness of using the component attributes in the overall attribute recognition process and further improves the accuracy of the overall attribute recognition result.

[0009] Further, in some embodiments, the component feature set, the component attribute recognition result set and the overall feature are fused to obtain a first fused feature, including: fusing the component feature set and the overall feature to obtain a second fused feature; fusing the second fused feature and the component attribute recognition result set to obtain the first fused feature.

[0010] The above scheme can realize the fusion of component feature set, component attribute recognition result set and overall feature.

[0011] Further, in some embodiments, the component feature set includes several component features, and the component feature set and the overall feature are fused to obtain a second fused feature, including: fusing the several component features to obtain a first component fusion result; fusing the first component fusion result with the overall feature to obtain a second fused feature.

[0012] The above solution can realize the integration of component feature set and overall feature.

[0013] Further, in some embodiments, the component attribute recognition result set includes several component attribute recognition results, and the second fusion feature and the component attribute recognition result set are fused to obtain the first fusion feature, including: fusing the several component attribute recognition results to obtain the second component fusion result; fusing the second fusion feature and the second component fusion result to obtain the first fusion feature.

[0014] The above solution can realize the fusion of the first fusion feature and the component attribute recognition result set.

[0015] Furthermore, in some embodiments, the first component fusion result is fused with the overall feature to obtain a second fusion feature, including: obtaining a correlation coefficient matrix between the first component fusion result and the overall feature; multiplying the first component fusion result, the overall feature, and the correlation coefficient matrix respectively to obtain a first multiplication result and a second multiplication result; adding the first multiplication result and the second multiplication result to obtain a second fusion feature.

[0016] The above scheme introduces the correlation coefficient matrix into the fusion process of the first component fusion result and the overall feature, which can improve the effectiveness of the fusion and further enhance the expressiveness of the second fusion feature on the overall attributes of the target to be identified.

[0017] Furthermore, in some embodiments, a component feature set, a component attribute recognition result set and an overall feature of a target to be identified in an image to be identified are obtained, including: performing short-distance dependent feature extraction on the image to be identified using a short-distance dependent feature extraction network to obtain a component feature set; and performing long-distance dependent feature extraction on the image to be identified using a long-distance dependent feature extraction network to obtain an overall feature.

[0018] In the above scheme, the component feature set is a short-distance dependent feature, and the overall feature is a long-distance dependent feature, which can improve the accuracy of the component attribute recognition result set and the subsequent overall attribute recognition result.

[0019] Furthermore, in some embodiments, obtaining a component feature set, a component attribute recognition result set and an overall feature of a target to be recognized in an image to be recognized also includes: performing component attribute recognition on the component feature set to obtain a component attribute recognition result set.

[0020] Compared with the method of performing component attribute recognition on the overall feature, the above scheme performs component attribute recognition on the component feature set, which can eliminate the interference between different component features and obtain a component attribute recognition result set of the target to be recognized with higher accuracy. In addition, the component attribute recognition result set can be applied to the fusion with the overall feature to improve the accuracy of overall attribute recognition.

[0021] Further, in some embodiments, the short-range dependency feature extraction network is a convolutional neural network.

[0022] Further, in some embodiments, the long-distance dependency feature extraction network is a neural network based on an attention mechanism.

[0023] Furthermore, in some embodiments, the component feature set and the component attribute recognition result set are obtained based on the component feature extraction module and the component attribute recognition module of the target attribute recognition network, respectively, and the training steps of the target attribute recognition network include: using the component feature extraction module to extract the training component feature set of the training target in the training image; using the component feature extraction module to perform component attribute recognition on the training target based on the training component feature set to obtain a training component attribute recognition result set; and adjusting the parameters of the component feature extraction module and the component attribute recognition module based on the training component attribute recognition result set.

[0024] The above scheme provides a target attribute recognition network, which can implement the above target attribute recognition method.

[0025] Further, in some embodiments, the target attribute recognition method is implemented based on a target attribute recognition network, the target attribute recognition network includes a component feature extraction module, a component attribute recognition module, an overall feature extraction module, a feature fusion module, and an overall attribute recognition module, and the target attribute recognition method includes: using the component feature extraction module to extract component features of the image to be recognized to obtain a component feature set; and using the overall feature extraction module to extract overall features of the image to be recognized to obtain overall features; and using the component attribute recognition module to perform component attribute recognition on the component feature set to obtain a component attribute recognition result set; using the feature fusion module to fuse at least one of the component feature set and the component attribute recognition result set with the overall feature to obtain a first fused feature; using the overall attribute recognition module to perform overall attribute recognition on the target to be recognized based on the first fused feature to obtain an overall attribute recognition result.

[0026] The above scheme adjusts the parameters of the component feature extraction module and the component attribute recognition module in the target attribute recognition network based on the training component attribute recognition result set. Therefore, there is no need to annotate additional supervisory information for the training of the component feature extraction module, and there is no need to re-annotate the supervisory information to adapt to different application scenarios. The training of the component feature extraction module can be achieved, so that the component feature extraction module can adaptively extract component features according to different components in the application stage and each application scenario, alleviate or even eliminate the interference between the features of each component, and improve the accuracy of component feature extraction. Therefore, while reducing the annotation cost and time cost in the training stage, it can also improve the accuracy of component feature extraction.

[0027] The present application provides a target attribute recognition device, including: an acquisition module, a fusion module and an attribute recognition module. The acquisition module is used to acquire a component feature set, a component attribute recognition result set and an overall feature of a target to be recognized in an image to be recognized; the fusion module is used to fuse at least one of the component feature set and the component attribute recognition result set with the overall feature to obtain a first fusion feature; the attribute recognition module is used to perform overall attribute recognition on the target to be recognized based on the first fusion feature to obtain an overall attribute recognition result.

[0028] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned target attribute recognition method.

[0029] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the above-mentioned target attribute identification method is implemented.

[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0032] Figure 1 It is a flowchart of an embodiment of a target attribute identification method provided by the present application;

[0033] Figure 2 It is a schematic diagram of the structure of the target attribute recognition network of this application;

[0034] Figure 3 It is a structural diagram of the component feature extraction module of this application;

[0035] Figure 4 It is a schematic diagram of the structure of the convolution module of this application;

[0036] Figure 5 It is a schematic diagram of the processing process of the semantic prediction module of this application;

[0037] Figure 6 This is a schematic diagram of Inception_a of this application;

[0038] Figure 7 This is a schematic diagram of Inception_b of this application;

[0039] Figure 8 This is a schematic diagram of Inception_c of this application;

[0040] Fig. 9 This is a schematic diagram of the processing of the first feature fusion module of this application;

[0041] Fig.10 This is a schematic diagram of the processing of the second feature fusion module of the present application;

[0042] Fig.11 It is a schematic diagram of the target attribute recognition network processing of this application;

[0043] Fig.12 It is a flowchart of another embodiment of the target attribute identification method provided by the present application;

[0044] Fig.13 It is a structural schematic diagram of an embodiment of a target attribute recognition device provided by the present application;

[0045] Fig.14 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0046] Fig.15 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0047] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0048] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0049] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0050] Figure 1 1 is a flow chart of an embodiment of a target attribute identification method provided by the present application. Figure 1 As shown, in this embodiment, the target attribute recognition method may include the following steps:

[0051] S110: Obtain a component feature set, a component attribute recognition result set and an overall feature of the target to be identified in the image to be identified.

[0052] The execution subject of this embodiment is a target attribute recognition device, and the target attribute recognition device can be any electronic device with target attribute recognition capability.

[0053] The image to be identified is an image containing an object to be identified.

[0054] The target to be identified can be any object with attributes that can be identified. Objects can include static objects and dynamic objects, such as vehicles, animals, and plants.

[0055] The overall feature is used to characterize the entire target to be identified. For example, the overall feature of a vehicle is used to characterize the vehicle as a whole. The overall feature of a pig is used to characterize the pig as a whole. The overall feature of a tree is used to characterize the tree as a whole.

[0056] The target to be identified includes several parts. For example, a vehicle includes the front, rear, roof, doors, and wheels. A pig includes the head, body, legs, and tail. A tree includes roots, trunks, branches, and leaves.

[0057] The component feature set includes several component features. A component feature is used to characterize a component of the target to be identified and at least one attribute of the component. For example, the front feature, rear feature, roof feature, door feature, and wheel feature of a vehicle respectively characterize the front, rear, roof, door, and wheel. The head feature, body feature, leg feature, and tail feature of a pig respectively characterize the head, body, leg, and tail. The root feature, trunk feature, branch feature, and leaf feature of a tree respectively characterize the root, trunk, branch, and leaf. Several component features include at least one component feature.

[0058] The component attribute recognition result set includes several component attribute recognition results. A component attribute recognition result represents at least one attribute of a component of the target to be recognized. The component attribute recognition results in the component attribute recognition result set are obtained by performing component attribute recognition on component features in the component feature set. For example, the leaf attribute recognition result obtained by performing leaf attribute recognition on leaf features represents the shape of the leaf. Several component attribute recognition results include at least one component attribute recognition result.

[0059] The component feature extraction network and the overall feature extraction network can be used to extract the component feature set and the overall feature respectively. The component feature extraction network and the overall feature extraction network are neural networks with feature extraction capabilities, and can be the same or different.

[0060] In some embodiments, one of the component feature extraction network and the overall feature extraction network is a long-distance dependency feature extraction network, and the other is a short-distance dependency feature extraction network. The features extracted by the short-distance dependency feature extraction network are short-distance dependency features, which may be, but are not limited to, a convolutional neural network (CNN). The features extracted by the long-distance dependency feature network are long-distance dependency features, and the long-distance dependency feature network may be, but are not limited to, a neural network (Transformer) based on an attention mechanism, or a non-local network. In some embodiments, the short-distance dependency feature extraction network is a convolutional neural network, and / or the long-distance dependency feature extraction network is a neural network based on an attention mechanism.

[0061] In some embodiments, S110 may include: performing short-distance dependency feature extraction on the image to be identified using a short-distance dependency feature extraction network to obtain a component feature set and an overall feature.

[0062] In some embodiments, S110 may include: performing long-distance dependent feature extraction on the image to be identified using a long-distance dependent feature extraction network to obtain a component feature set and an overall feature.

[0063] In some embodiments, S110 may include: using a long-distance dependent feature extraction network to extract long-distance dependent features from the image to be identified to obtain a component feature set; and using a short-distance dependent feature extraction network to extract short-distance dependent features from the image to be identified to obtain an overall feature.

[0064] In some embodiments, S110 may include: using a short-distance dependent feature extraction network to perform short-distance dependent feature extraction on the image to be identified to obtain a component feature set; and using a long-distance dependent feature extraction network to perform long-distance dependent feature extraction on the image to be identified to obtain an overall feature. It can be understood that the short-distance dependent feature can more accurately represent the local semantic information of the image, and the long-distance dependent feature can more accurately represent the global semantic information of the image. Therefore, the component feature set obtained by the short-distance dependent feature extraction can more accurately represent the components of the target to be identified. And the overall feature obtained by the long-distance dependent feature extraction can more accurately represent the target to be identified as a whole. And the long-distance dependent feature and the short-distance dependent feature can complement each other. Therefore, the component feature set is a short-distance dependent feature, and the overall feature is a long-distance dependent feature, which can improve the accuracy of the component attribute recognition result set and the subsequent overall attribute recognition results.

[0065] In some embodiments, component attribute recognition can be performed on a component feature set to obtain a component attribute recognition result set. It is understandable that, compared with the method of performing component attribute recognition on the overall feature, performing component attribute recognition on a component feature set can eliminate interference between different component features, and obtain a component attribute recognition result set of the target to be recognized with higher accuracy.

[0066] S120: Fusing at least one of the component feature set and the component attribute recognition result set with the overall feature to obtain a first fused feature.

[0067] In S120, the feature fusion method includes but is not limited to an addition operation, a multiplication operation, a concatenation operation, and a weighted average operation.

[0068] In some embodiments, S120 may include S121 (not shown). S121: Fusing the component feature set, the component attribute recognition result set, and the overall feature to obtain a first fused feature.

[0069] In some embodiments, S121 may include S1211-S1212 (not shown). S1211: Fusing the component feature set and the overall feature to obtain a second fused feature. S1212: Fusing the second fused feature and the component attribute recognition result set to obtain a first fused feature.

[0070] In some embodiments, S1211 may include S12111-S12112 (not shown). S12111: Fusing several component features to obtain a first component fusion result. S12112: Fusing the first component fusion result with the overall feature to obtain a second fusion feature.

[0071] In some embodiments, S12112 may include: directly fusing the first component fusion result with the overall feature to obtain a first fusion feature.

[0072] In some embodiments, S12112 may include: obtaining a correlation coefficient matrix between the first component fusion result and the overall feature; respectively multiplying the first component fusion result, the overall feature and the correlation coefficient matrix to obtain a first multiplication result and a second multiplication result; adding the first multiplication result and the second multiplication result to obtain a second fusion feature. Among them, the first component fusion result and the overall feature may be feature spliced ​​to obtain a feature splicing result; the feature splicing result may be convolved, pooled, and normalized to obtain a correlation coefficient matrix. It can be understood that the correlation coefficient matrix represents the correlation between the fusion component feature and the corresponding feature point in the overall feature. The greater the correlation, the greater the corresponding correlation coefficient in the correlation coefficient matrix. Therefore, introducing the correlation coefficient matrix in the fusion process of the first component fusion result and the overall feature can improve the effectiveness of the fusion, thereby improving the expressiveness of the second fusion feature on the overall attributes of the target to be identified.

[0073] In some embodiments, S1211 may include S12113-S12114 (not shown). S12113: Fusing several component features with the overall feature to obtain several second fused sub-features. S12114: Fusing several second fused sub-features to obtain a second fused feature.

[0074] In some embodiments, S1212 may include S12121-S12122 (not shown). S12121: Fusing several component attribute recognition results to obtain a second component fusion result. S12122: Fusing the second fusion feature and the second component fusion result to obtain a first fusion feature.

[0075] In some embodiments, S1212 may include S12123-S12124 (not shown). S12123: Fusing several component attribute recognition results with the second fusion feature respectively to obtain several first fusion sub-features. S12124: Fusing several first fusion sub-features to obtain the first fusion feature.

[0076] In some embodiments, S120 may include S122 (not shown). S122: Fusing the component feature set and the overall feature to obtain a first fused feature. S122 is similar to S1211, please refer to the detailed description of S1211, which will not be repeated here.

[0077] In some embodiments, S120 may include S123 (not shown). S123: Fusing the component attribute recognition result set and the overall feature to obtain a first fused feature.

[0078] In some embodiments, S123 may include S1231-S1232 (not shown). S1231: Fusing several component attribute recognition results to obtain a third component fusion result. S1232: Fusing the third component fusion result with the overall feature to obtain a first fusion feature.

[0079] In some embodiments, S123 may include S1233-S1234 (not shown). S1233: Fusing several component attribute recognition results with the overall feature respectively to obtain several first fused sub-features. S1214: Fusing several first fused sub-features to obtain a first fused feature.

[0080] It can be understood that, regarding the above S121-S123, since S121 integrates the component attribute recognition result set and the component feature set with the overall features compared to S122 and S123, the overall attribute recognition process can better capture the relationship between component attributes and overall attributes, further improve the effectiveness of utilizing component attributes in the overall attribute recognition process, and further improve the accuracy of the overall attribute recognition results.

[0081] S130: Performing overall attribute recognition on the target to be recognized based on the first fusion feature to obtain an overall attribute recognition result.

[0082] The overall attribute recognition result represents the overall attribute of the target to be recognized. For example, the overall attributes of a vehicle include the model and direction, the overall attributes of a pig include the pig's movements, behavior, and breed, and the overall attributes of a tree include the tree's breed.

[0083] Through the implementation of this embodiment (S110-S130), a component feature set, a component attribute recognition result set and an overall feature in the image to be recognized are obtained; at least one of the component feature set and the component attribute recognition result set is fused with the overall feature to obtain a first fused feature and an overall attribute recognition result. The component feature set and the component attribute recognition result set represent the component attributes, the overall feature represents the overall attribute, and the overall attribute is associated with the component attribute. Therefore, compared with overall attribute recognition based only on overall features, in this embodiment, the first fused feature obtained by fusing the component feature set / component attribute recognition result and the overall feature is used for overall attribute recognition, and the correlation between the component attributes and the overall attributes is effectively used in the overall attribute recognition process, so that the component attributes guide the overall attribute recognition process, thereby improving the accuracy of the overall attribute recognition result.

[0084] Further, in some embodiments, the target attribute recognition method is implemented based on a target attribute recognition network. The target attribute recognition network includes a feature extraction module and an attribute recognition module. The feature extraction module includes a component feature extraction module and an overall feature extraction module, and the attribute recognition module includes a component attribute recognition module, a feature fusion module and an overall attribute recognition module. In some embodiments, one of the component feature extraction module and the overall feature extraction module can be the aforementioned short-distance dependency feature extraction network, and the other can be the aforementioned long-distance dependency feature extraction network. The output end of at least one of the component feature extraction module and the component attribute recognition module is connected to the input end of the feature fusion module. Figure 2 It is a schematic diagram of the structure of the target attribute recognition network of this application, such as Figure 2 As shown, the output ends of the component feature extraction module and the component attribute recognition module are both connected to the input end of the feature fusion module.

[0085] In S110, the component feature extraction module can be used to extract component features of the image to be identified to obtain a component feature set; the overall feature extraction module can be used to extract overall features of the image to be identified to obtain overall features; and the component attribute recognition module can be used to recognize component attributes of the component feature set to obtain a component attribute recognition result set.

[0086] In S120, a feature fusion module may be used to fuse at least one of the component feature set and the component attribute recognition result set with the overall feature to obtain a first fused feature;

[0087] In S130, the overall attribute recognition module may be used to perform overall attribute recognition on the target to be recognized based on the first fusion feature to obtain an overall attribute recognition result.

[0088] For ease of understanding, some modules in the target attribute recognition network are illustrated as follows. It should be noted that the following modules are intended to be examples and do not limit the structure of the target attribute recognition network.

[0089] (1) About the component feature extraction module

[0090] As mentioned above, the component feature extraction module can be a convolutional neural network (CNN), a neural network based on an attention mechanism (Transformer), a non-local network, and the like.

[0091] In some embodiments, the component feature extraction module is a convolutional neural network (CNN), and the component feature extraction module includes a convolution module and a semantic prediction module. The image to be identified is input into the component feature extraction module.

[0092] In some embodiments, the component feature extraction module is a convolutional neural network (CNN), and the component feature extraction module includes a convolution module, a semantic prediction module, and an InceptionV3 module. The InceptionV3 module can extract deeper semantic information based on the semantic prediction module, which helps to improve the expressiveness of the component feature set extracted by the component feature extraction module on the component of the target to be identified, thereby improving the accuracy of the target attribute recognition result.

[0093] Combine as follows Figure 3-Figure 10 The component feature extraction module is described.

[0094] Figure 3 It is a schematic diagram of the structure of the component feature extraction module of this application. Figure 3 As shown in the figure, the component feature extraction module includes a convolution module, a semantic prediction module and an InceptionV3 module.

[0095] Figure 4 It is a schematic diagram of the structure of the convolution module of this application. Figure 4 As shown in the figure, the convolution module includes a convolution layer (Conv), a batch normalization layer (BN), and an activation layer (Relu). The processing process of the convolution module includes: inputting the 3-channel image to be recognized into the convolution module, and processing it in sequence by 3×3Conv, BN, Relu, 3×3Conv, BN, and Relu. The number of channels of the image to be recognized in the above layers is changed to 64, 64, 128, 128, 128, and 128, respectively, and a B×C×W×H feature map with a size of 1 / 2 of the image to be recognized and a channel number of 128 is obtained. The B×C×W×H feature map obtained by the convolution module is sent to the semantic prediction module.

[0096] The semantic prediction module includes convolutional layers and activation layers. The convolutional layer uses a convolution kernel of size 5×5, stride 1, and dilation 2, which does not change the resolution of the input feature map.

[0097] Figure 5 Schematic diagram of the processing of the semantic prediction module of this application. Figure 5 As shown in the figure, in the convolution layer, the B×C×W×H feature map is processed to obtain a B×N×W×H semantic feature tensor. B represents the number of images to be recognized. N represents the number of components that the semantic prediction module can predict (may include or exclude the background). For example, N=4, the components include roots, trunks, branches, and leaves. For another example, N=5, the components include roots, trunks, branches, leaves, and background. W and H represent the height and width of the feature map, respectively. The B×N×W×H semantic feature tensor can be expanded according to the component dimension to obtain N B×1×W×H semantic feature maps. When the number of components includes the background, the semantic feature map representing the background can be removed, and only N-1 semantic feature maps outside the background are retained. In the activation layer, each expanded semantic feature map (B,1,W,H …B,N / 2,W,H…B,N,W,H) is multiplied by the B×C×W×H feature map to obtain each initial component feature map B×C×W×H.

[0098] Figure 6-Figure 8 This is a schematic diagram of the structure of the InceptionV3 module of this application. The InceptionV3 module includes three different Inception modules (Inception_a, Inception_b, Inception_c). Figure 6 This is a schematic diagram of Inception_a of this application. Figure 7 This is a schematic diagram of Inception_b of this application. Figure 8 This is a schematic diagram of Inception_c of this application. The structures of Inception_a, Inception_b, and Inception_c are prior art. Figure 6-Figure 8 The structure shown in the figure is sufficient and will not be described in detail here. After the InceptionV3 module processes each initial component feature map B×C×W×H, the final component feature map can be obtained.

[0099] (2) About the overall feature extraction module

[0100] As mentioned above, the overall feature extraction module can be a convolutional neural network (CNN), a neural network based on an attention mechanism (Transformer), a non-local network, etc. For example, the overall feature extraction module is a Transformer, and the structure of the Transformer is an existing technology and will not be described here.

[0101] (3) About feature fusion module

[0102] The feature fusion module includes a first feature fusion module and a second feature fusion module.

[0103] In S120, the component feature set and the overall feature may be fused using a first feature fusion module to obtain a second fused feature; and the second fused feature and the component attribute recognition result set may be fused using a second feature fusion module to obtain a first fused feature.

[0104] ①The first feature fusion module

[0105] Fig. 9 This is a schematic diagram of the processing of the first feature fusion module of this application. Fig. 9 As shown in the figure, the component feature set (CNN feature) is redefined to the same dimension as the overall feature (Transformer feature) to obtain the component feature set (CNN feature, C j ×H×W) and overall features (Transformer features, C i ×H×W), and then perform feature concatenation to obtain (C j + C i )×H×W second fusion feature. j + C i )×H×W The second fusion feature is subjected to 5×5 convolution, 1×1 convolution, pooling (GAP), 1×1 convolution, and normalization (sigmoid) to obtain a correlation coefficient matrix. The component feature set and the overall feature are respectively multiplied element by element with the correlation coefficient matrix to obtain the first multiplication result and the second multiplication result. The first multiplication result and the second multiplication result are added element by element to obtain the fused feature (the second fusion feature).

[0106] ②The second feature fusion module

[0107] Fig.10 This is a schematic diagram of the second feature fusion module processing of this application. Fig.10 As shown, the second component fusion result and the second fusion feature obtained by fusing several component attribute recognition results in the component attribute recognition result set are input into the second feature fusion module. In the second feature fusion module, the second fusion feature (the feature after the fusion of CNN feature and Transformer feature) is subjected to the pooling layer (GAP) and the fully connected layer (FC) operation; then the operation result is concatenated (concat) and the fully connected layer (FC) operation with the second component fusion result (the fusion vector of the local attribute) to obtain the first fusion feature. Subsequently, the first fusion feature is subjected to the overall attribute recognition to obtain the overall attribute recognition result.

[0108] For ease of understanding, the following combination Fig.11 The target attribute recognition method based on the target attribute recognition network is explained in the form of an example.

[0109] Fig.11 This is a schematic diagram of the target attribute recognition network processing of this application. Fig.11 As shown in FIG, the target attribute recognition network includes a feature extraction module with semantic information (component feature extraction module), and an adaptive multi-level attribute output module integrating CNN and Transformer (including an overall feature extraction module and an attribute recognition module).

[0110] Input the image to be recognized into the target attribute recognition network.

[0111] In the feature extraction module with semantic information:

[0112] First, the convolution module and the unsupervised semantic prediction branch are used to process the image to be recognized to obtain the initial component 1 feature map, component 2 feature map, and component 3 feature map. Then, Inception_a, Inception_b, and Inception_c (not shown) are used to process the initial component 1 feature map, component 2 feature map, and component 3 feature map respectively to obtain the final component 1 feature map, component 2 feature map, and component 3 feature map.

[0113] In the adaptive multi-level attribute output module:

[0114] The attribute prediction branch (component attribute recognition module) is used to process the final component 1 feature map, component 2 feature map, and component 3 feature map to obtain component 1 attribute recognition results, component 2 attribute recognition results, and component 3 attribute recognition results.

[0115] The Transformer encoder (overall feature extraction module) is used to process the image to be identified and obtain the overall feature map.

[0116] The final component 1 feature map, component 2 feature map, component 3 feature map and overall feature map are fused using the CNN and Transformer feature fusion module (first feature fusion module) to obtain the second fused feature map.

[0117] The attribute fusion module (second feature fusion module) is used to fuse the attribute recognition result of component 1, the attribute recognition result of component 2, the attribute recognition result of component 3 and the second fused feature map to obtain a first fused feature map.

[0118] The overall attribute recognition module is used to perform overall attribute recognition on the first fused feature map to obtain an overall attribute recognition result.

[0119] Fig.12is a flow chart of another embodiment of the target attribute recognition method provided by the present application. In this embodiment, the component feature set and the component attribute recognition result set are obtained based on the component feature extraction module and the component attribute recognition module of the target attribute recognition network respectively. Fig.12 As shown, the training steps of the target attribute recognition network include:

[0120] S210: Extracting a training component feature set of a training target in a training image using a component feature extraction module.

[0121] The component feature extraction module can be the aforementioned short-distance dependency feature extraction network and long-distance dependency feature extraction network.

[0122] The training target can be any object that has attributes and can be recognized. It should be the same or similar to the target category to be recognized.

[0123] The training component feature set is similar to the component feature set of the target to be identified, and will not be described in detail here.

[0124] S220: Using the component feature extraction module to perform component attribute recognition on the training target based on the training component feature set, and obtaining a training component attribute recognition result set.

[0125] The attribute recognition result set of the training component is similar to the attribute recognition result set of the aforementioned target to be recognized, and will not be described in detail here.

[0126] S230: Adjusting parameters of the component feature extraction module and the component attribute recognition module based on the training component attribute recognition result set.

[0127] For other detailed descriptions of this embodiment (S210-S230), please refer to the previous embodiments and will not be repeated here.

[0128] The processing of the target attribute recognition network in the application phase and the training phase is similar.

[0129] In some embodiments, the training of other modules of the target attribute recognition network can be constrained by the training component attribute recognition results and the training overall attribute recognition results, which will not be elaborated here.

[0130] Through the implementation of this embodiment (S210-S230), the parameters of the component feature extraction module and the component attribute recognition module in the target attribute recognition network are adjusted based on the training component attribute recognition result set. Therefore, there is no need to annotate additional supervisory information for the training of the component feature extraction module, and there is no need to re-annotate supervisory information to adapt to different application scenarios. The training of the component feature extraction module can be achieved, so that the component feature extraction module can adaptively extract component features according to different components in the application stage and each application scenario, alleviate or even eliminate the interference between the features of each component, and improve the accuracy of component feature extraction. Therefore, while reducing the annotation cost and time cost in the training stage, it can also improve the accuracy of component feature extraction.

[0131] Fig.13 Schematic diagram of the structure of an embodiment of the target attribute recognition device provided by the present application. Fig.13 As shown, the target attribute recognition device 30 includes an acquisition module 31 , a fusion module 32 and an attribute recognition module 33 .

[0132] The acquisition module 31 is used to acquire a component feature set, a component attribute recognition result set and an overall feature of the target to be identified in the image to be identified;

[0133] A fusion module 32, used to fuse at least one of the component feature set and the component attribute recognition result set with the overall feature to obtain a first fused feature;

[0134] The attribute recognition module 33 is used to perform overall attribute recognition on the target to be recognized based on the first fusion feature to obtain an overall attribute recognition result.

[0135] Further, in some embodiments, in the fusion module 32, at least one of the component feature set and the component attribute recognition result set is fused with the overall feature to obtain a first fused feature, including: fusing the component feature set, the component attribute recognition result set and the overall feature to obtain a first fused feature.

[0136] Further, in some embodiments, in the fusion module 32, the component feature set, the component attribute recognition result set and the overall feature are fused to obtain a first fused feature, including: fusing the component feature set and the overall feature to obtain a second fused feature; fusing the second fused feature and the component attribute recognition result set to obtain the first fused feature.

[0137] Further, in some embodiments, in the fusion module 32, the component feature set includes several component features, and the component feature set and the overall feature are fused to obtain a second fused feature, including: fusing the several component features to obtain a first component fusion result; fusing the first component fusion result with the overall feature to obtain a second fused feature.

[0138] Further, in some embodiments, in the fusion module 32, the component attribute recognition result set includes several component attribute recognition results, and the second fusion feature and the component attribute recognition result set are fused to obtain the first fusion feature, including: fusing the several component attribute recognition results to obtain the second component fusion result; fusing the second fusion feature and the second component fusion result to obtain the first fusion feature.

[0139] Furthermore, in some embodiments, in the fusion module 32, the first component fusion result is fused with the overall feature to obtain a second fusion feature, including: obtaining a correlation coefficient matrix between the first component fusion result and the overall feature; multiplying the first component fusion result, the overall feature, and the correlation coefficient matrix respectively to obtain a first multiplication result and a second multiplication result; adding the first multiplication result and the second multiplication result to obtain a second fusion feature.

[0140] Furthermore, in some embodiments, in the acquisition module 31, a component feature set, a component attribute recognition result set and an overall feature of the target to be identified in the image to be identified are obtained, including: using a short-distance dependent feature extraction network to perform short-distance dependent feature extraction on the image to be identified to obtain a component feature set; and using a long-distance dependent feature extraction network to perform long-distance dependent feature extraction on the image to be identified to obtain an overall feature.

[0141] Furthermore, in some embodiments, in the acquisition module 31, a component feature set, a component attribute recognition result set and an overall feature of the target to be identified in the image to be identified are acquired, and also includes: performing component attribute recognition on the component feature set to obtain a component attribute recognition result set.

[0142] Further, in some embodiments, the short-range dependency feature extraction network is a convolutional neural network.

[0143] Further, in some embodiments, the long-distance dependency feature extraction network is a neural network based on an attention mechanism.

[0144] Furthermore, in some embodiments, the component feature set and the component attribute recognition result set are obtained based on the component feature extraction module and the component attribute recognition module of the target attribute recognition network, respectively, and the training steps of the target attribute recognition network include: using the component feature extraction module to extract the training component feature set of the training target in the training image; using the component feature extraction module to perform component attribute recognition on the training target based on the training component feature set to obtain a training component attribute recognition result set; and adjusting the parameters of the component feature extraction module and the component attribute recognition module based on the training component attribute recognition result set.

[0145] Further, in some embodiments, the target attribute recognition method is implemented based on a target attribute recognition network, the target attribute recognition network includes a component feature extraction module, a component attribute recognition module, an overall feature extraction module, a feature fusion module, and an overall attribute recognition module, and the target attribute recognition method includes: using the component feature extraction module to extract component features of the image to be recognized to obtain a component feature set; and using the overall feature extraction module to extract overall features of the image to be recognized to obtain overall features; and using the component attribute recognition module to perform component attribute recognition on the component feature set to obtain a component attribute recognition result set; using the feature fusion module to fuse at least one of the component feature set and the component attribute recognition result set with the overall feature to obtain a first fused feature; using the overall attribute recognition module to perform overall attribute recognition on the target to be recognized based on the first fused feature to obtain an overall attribute recognition result.

[0146] Fig.14 Schematic diagram of the structure of an embodiment of the electronic device of the present application. Fig.14 As shown, the electronic device 40 includes a memory 41 and a processor 42, and the processor 42 is used to execute the program instructions stored in the memory 41 to implement the steps in any of the above method embodiments. In a specific implementation scenario, the electronic device 40 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 40 may also include a carrier device such as a laptop computer and a tablet computer, which is not limited here.

[0147] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0148] See also Fig.15 , Fig.15 The computer-readable storage medium 50 stores program instructions 51, which implement the steps in any of the above method embodiments when executed by a processor.

[0149] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0150] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0151] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. In another image position, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0152] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A target attribute recognition method, characterized in that: include: Acquire a component feature set, a component attribute recognition result set, and an overall feature of a target to be recognized in a target image to be recognized, wherein the component feature set includes a plurality of component features, the component attribute recognition result set includes a plurality of component attribute recognition results respectively corresponding to the plurality of component features, the component attribute recognition results are obtained by performing component attribute recognition based on the corresponding component features, the component feature represents a component of the target to be recognized, and the overall feature represents the target to be recognized as a whole; Fusing the component feature set, the component attribute recognition result set, and the overall feature to obtain a first fused feature; Based on the first fusion feature, overall attribute recognition is performed on the target to be recognized to obtain an overall attribute recognition result.

2. The method according to claim 1, characterized in that The fusing the component feature set, the component attribute recognition result set and the overall feature to obtain a first fused feature includes: Fusing the component feature set and the overall feature to obtain a second fused feature; The second fused feature and the component attribute recognition result set are fused to obtain the first fused feature.

3. The method according to claim 2, characterized in that The step of fusing the component feature set and the overall feature to obtain a second fused feature includes: Fusing the features of several components to obtain a first component fusion result; Fusing the first component fusion result with the overall feature to obtain the second fusion feature; And / or, fusing the second fused feature with the component attribute recognition result set to obtain the first fused feature, including: Fusing the plurality of component attribute recognition results to obtain a second component fusion result; The second fusion feature and the second component fusion result are fused to obtain the first fusion feature.

4. The method according to claim 3, characterized in that The step of fusing the first component fusion result with the overall feature to obtain the second fusion feature includes: Obtaining a correlation coefficient matrix between the first component fusion result and the overall feature; Respectively multiplying the first component fusion result, the overall feature and the correlation coefficient matrix to obtain a first multiplication result and a second multiplication result; The first multiplication result and the second multiplication result are added to obtain the second fusion feature.

5. The method according to claim 1, characterized in that The step of obtaining a component feature set, a component attribute recognition result set and an overall feature of a target to be identified in an image to be identified includes: Using a short-distance dependency feature extraction network to extract short-distance dependency features from the image to be identified, to obtain the component feature set; and A long-distance dependent feature extraction network is used to extract long-distance dependent features of the image to be identified to obtain the overall features.

6. The method according to claim 5, characterized in that The short-distance dependency feature extraction network is a convolutional neural network, and / or the long-distance dependency feature extraction network is a neural network based on an attention mechanism.

7. The method according to claim 1, characterized in that The component feature set and the component attribute recognition result set are obtained based on the component feature extraction module and the component attribute recognition module of the target attribute recognition network, respectively. The training steps of the target attribute recognition network include: Extracting a training component feature set of a training target in a training image using the component feature extraction module; Using the component attribute recognition module to recognize component attributes of the training target based on the training component feature set, to obtain a training component attribute recognition result set; Based on the training component attribute recognition result set, the parameters of the component feature extraction module and the component attribute recognition module are adjusted.

8. The method according to claim 1, characterized in that The target attribute recognition method is implemented based on a target attribute recognition network, the target attribute recognition network includes a component feature extraction module, a component attribute recognition module, an overall feature extraction module, a feature fusion module, and an overall attribute recognition module. The target attribute recognition method includes: Using the component feature extraction module to extract component features from the image to be identified, and obtaining the component feature set; using the overall feature extraction module to extract overall features from the image to be identified, and obtaining the overall features; and using the component attribute recognition module to recognize component attributes from the component feature set, and obtaining the component attribute recognition result set; Using the feature fusion module to fuse the component feature set, the component attribute recognition result set and the overall feature to obtain the first fused feature; The overall attribute recognition module is used to perform overall attribute recognition on the target to be recognized based on the first fusion feature to obtain the overall attribute recognition result.

9. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images