Multi-attribute recognition method, electronic device, and computer-readable storage medium

By acquiring the features of the image to be identified and the pre-configured text feature set, the matching attribute combination is determined, which solves the problem of low accuracy of multi-attribute recognition and improves the accuracy and adaptability of the recognition.

CN118644855BActive Publication Date: 2024-11-08ZHEJIANG DAHUA TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411130608.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-08
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

The prior art has low accuracy in multi-attribute recognition and is difficult to effectively improve.

Method used

By obtaining the features of the image to be identified and the pre-configured text feature set, the text feature set consists of description text and attribute features of multiple attribute combinations, and the matching attribute combination is determined based on these features to improve the accuracy of multi-attribute recognition.

Benefits of technology

The accuracy of multi-attribute recognition is improved, and the attribute combination of images to be identified is determined by more matching the reference text features of the multi-attribute recognition requirements, which enhances the adaptability and accuracy of the recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118644855B_ABST
    Figure CN118644855B_ABST
Patent Text Reader

Abstract

The present application discloses a multi-attribute recognition method, an electronic device and a computer-readable storage medium, the method comprising: obtaining the image features to be recognized corresponding to the image to be recognized, and a text feature set configured for the image to be recognized; wherein the text feature set is obtained based on the following steps, including: obtaining the description texts corresponding to each of the multiple attribute combinations, and the attribute features corresponding to each attribute combination; based on the description text corresponding to each attribute combination and the attribute features corresponding to each attribute combination, determining the reference text features corresponding to each attribute combination, and obtaining a text feature set composed of the reference text features corresponding to each of the multiple attribute combinations; the description text includes the multiple attributes included in the attribute combination and the description content corresponding to each attribute; based on the image features to be recognized and the text feature set, determining the attribute combination matched by the image to be recognized. In the above manner, the present application can improve the accuracy of multi-attribute recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multi-attribute recognition method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the advent of the data age, a large number of images need to be processed, and how to identify the attributes of images has become an important direction. In the prior art, attribute recognition models are usually trained to identify the attributes of images, but conventional attribute recognition models have a better accuracy rate when performing single attribute recognition, but a lower accuracy rate when performing multi-attribute recognition. In view of this, how to improve the accuracy of multi-attribute recognition has become a problem that needs to be solved urgently. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a multi-attribute recognition method, an electronic device and a computer-readable storage medium, which can improve the accuracy of multi-attribute recognition.

[0004] To solve the above technical problems, the first aspect of the present application provides a multi-attribute recognition method, comprising: obtaining image features to be recognized corresponding to the image to be recognized, and a text feature set configured for the image to be recognized; wherein the text feature set is configured based on the following steps, comprising: obtaining description texts corresponding to each of a plurality of attribute combinations, and attribute features corresponding to each of the attribute combinations; based on the description text corresponding to each of the attribute combinations and the attribute features corresponding to each of the attribute combinations, determining the reference text features corresponding to each of the attribute combinations, and obtaining the text feature set composed of the reference text features corresponding to each of the attribute combinations; the description text includes the multiple attributes included in the attribute combination and the description content corresponding to each of the attributes; based on the image features to be recognized and the text feature set, determining the attribute combination matched by the image to be recognized.

[0005] To solve the above technical problem, the second aspect of the present application provides an electronic device, which includes: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method described in the first aspect.

[0006] In order to solve the above technical problem, the third aspect of the present application provides a computer-readable storage medium on which program data is stored. When the program data is executed by a processor, the method described in the first aspect is implemented.

[0007] The above scheme obtains the image features to be identified corresponding to the image to be identified, and a text feature set composed of multiple reference text features configured for the image to be identified, wherein the text feature set is pre-configured, and the configuration process includes: obtaining the description texts corresponding to each of the multiple attribute combinations, and obtaining the attribute features corresponding to each attribute combination, determining the reference text features corresponding to the attribute combination based on the description text corresponding to each attribute combination and the attribute features corresponding to each attribute combination, and combining the reference text features corresponding to all attribute combinations to obtain the text feature set. Among them, the attribute combination corresponds to multiple attributes, and the description text includes multiple attributes of the corresponding attribute combination and the description content used to describe each attribute. Therefore, the reference text features obtained based on the description text corresponding to each attribute combination have higher adaptability to multi-attribute recognition, and the reference text features also incorporate the attribute features of the corresponding attribute combination, so as to better meet the feature requirements for multi-attribute recognition. Based on the features of the image to be identified and multiple reference text features in the text feature set, the reference text features that match the image to be identified are determined, so that the attribute combination matched by the image to be identified is determined using the reference text features that better meet the needs of multi-attribute recognition, and multiple attributes matched by the image to be identified are obtained, thereby improving the accuracy of multi-attribute recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0009] Figure 1 It is a flowchart of an implementation method of the multi-attribute recognition method of the present application;

[0010] Figure 2 It is a flowchart of another implementation method of the multi-attribute recognition method of the present application;

[0011] Figure 3 It is a schematic diagram of an application scenario of an implementation method of a feature 1 of the reference text of this application;

[0012] Figure 4 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0013] Figure 5 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different implementation methods can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0015] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0016] The multi-attribute recognition method provided in the present application is used to recognize the attributes of an image, and its corresponding execution subject is a terminal capable of performing image processing.

[0017] See also Figure 1 , Figure 1 : is a flowchart of an implementation method of a multi-attribute recognition method of the present application, the method comprising:

[0018] S101: Obtain image features to be identified corresponding to the image to be identified, and a text feature set configured for the image to be identified; wherein the text feature set is configured based on the following steps, including: obtaining description texts corresponding to each of a plurality of attribute combinations, and attribute features corresponding to each attribute combination; based on the description text corresponding to each attribute combination and the attribute features corresponding to each attribute combination, determining a reference text feature corresponding to each attribute combination, and obtaining a text feature set composed of reference text features corresponding to each of the plurality of attribute combinations; the description text includes a plurality of attributes included in the attribute combination and a description content corresponding to each attribute.

[0019] Specifically, the image features to be recognized corresponding to the image to be recognized and a text feature set composed of a plurality of reference text features configured for the image to be recognized are obtained.

[0020] It should be noted that the text feature set is pre-configured, and the configuration process includes: obtaining the description texts corresponding to multiple attribute combinations, and obtaining the attribute features corresponding to each attribute combination, determining the reference text features corresponding to the attribute combination based on the description text corresponding to each attribute combination and the attribute features corresponding to each attribute combination, and combining the reference text features corresponding to all attribute combinations to obtain the text feature set.

[0021] It can be understood that there are multiple attributes corresponding to an attribute combination, and the description text includes multiple attributes of the corresponding attribute combination and description content used to describe each attribute. Therefore, the reference text features obtained based on the description text corresponding to each attribute combination have higher adaptability to multi-attribute recognition, and the reference text features also incorporate the attribute features of the corresponding attribute combination, thereby better meeting the feature requirements for multi-attribute recognition.

[0022] In some implementation scenarios, an image to be identified is obtained, and the image to be identified is input into an image encoder to obtain features of the image to be identified, and a text feature set including multiple reference text features preset in the text encoder is obtained. The reference text features are generated by pairing a reference vector with an attribute feature, and the reference text features are obtained in the process of training the image encoder and the text encoder. The training data used to train the image encoder and the text encoder include description texts matched by multiple attribute combinations, so as to obtain reference vectors that are more in line with the needs of multi-attribute recognition. The description texts corresponding to all attribute combinations are input into the text encoder during the training process, and after the training is completed, the description texts corresponding to all attribute combinations converge into a reference vector, and a single reference vector is paired with the attribute features of each attribute combination to obtain a reference text feature, and all reference text features together constitute a text feature set.

[0023] It can be understood that based on the description text corresponding to each attribute combination and the attribute features corresponding to each attribute combination, the reference text features corresponding to each attribute combination are determined, including: based on the description texts corresponding to all attribute combinations, a reference vector is obtained, and the reference vector is paired with the attribute features corresponding to each attribute combination to obtain the reference text features.

[0024] In some implementation scenarios, an image to be identified is obtained, and feature extraction is performed on the image to be identified to obtain features of the image to be identified, and a text feature set including multiple reference text features pre-configured for the image to be identified is obtained. The reference text features are generated by pairing global text features and attribute features, wherein the global text features are obtained in the process of training the image-text recognition model, and the training data used to train the image-text recognition model includes description texts matched by multiple attribute combinations, so as to obtain global text features with wider adaptability. The global text features corresponding to each attribute combination are integrated with features extracted from description texts of other attribute combinations during the training process, and after the training is completed, the description text corresponding to each attribute combination is matched with a global text feature, and for each attribute combination, the global text features are paired with the attribute features to obtain reference text features, and all reference text features together constitute a text feature set.

[0025] It can be understood that based on the description text corresponding to each attribute combination and the attribute features corresponding to each attribute combination, the reference text features corresponding to each attribute combination are determined, including: based on the description texts corresponding to all attribute combinations, the global text features corresponding to each attribute combination are obtained, and for each attribute combination, the global text features and the attribute features are paired to obtain the reference text features.

[0026] S102: Determine the attribute combination that matches the image to be identified based on the image features to be identified and the text feature set.

[0027] Specifically, based on the image features to be identified and multiple reference text features in the text feature set, the reference text features that match the image to be identified are determined, so that the attribute combination matched by the image to be identified is determined by using the reference text features that better meet the needs of multi-attribute recognition, and multiple attributes matched by the image to be identified are obtained, thereby improving the accuracy of multi-attribute recognition.

[0028] In some implementation scenarios, the similarity between the image feature to be identified and each reference text feature in the text feature set is obtained, and the reference text feature with the highest similarity is used as the reference text feature for matching the image to be identified, and the attribute combination corresponding to the reference text feature matched by the image to be identified is used as the attribute combination matched by the image to be identified.

[0029] In some implementation scenarios, among all the reference text features corresponding to the text feature set, at least one reference text feature that satisfies the matching condition with the image feature to be identified is determined, the matching confidences of all the reference text features that meet the matching condition and the image feature to be identified are obtained, and the attribute combination corresponding to the reference text feature with the highest matching confidence is used as the attribute combination matched by the image to be identified. Among them, the matching condition is related to the texture features contained in the image feature to be identified and the reference text feature, and the matching confidence is related to the semantic features contained in the image feature to be identified and the reference text feature, so that some reference text features are screened through shallow texture features, and then the reference text features with the highest matching degree with the image feature to be identified are screened through deep semantic features.

[0030] It can be understood that the image to be identified is an image including a target, and the attribute combination matched by the image to be identified corresponds to the multiple attributes corresponding to the target in the image to be identified. When the acquired original image includes multiple targets to be identified, the image including the target is extracted and converted to the same size to obtain the image to be identified, thereby ensuring that the attributes of the target in the image to be identified can be stably identified.

[0031] The above scheme obtains the image features to be identified corresponding to the image to be identified, and a text feature set composed of multiple reference text features configured for the image to be identified, wherein the text feature set is pre-configured, and the configuration process includes: obtaining the description texts corresponding to each of the multiple attribute combinations, and obtaining the attribute features corresponding to each attribute combination, determining the reference text features corresponding to the attribute combination based on the description text corresponding to each attribute combination and the attribute features corresponding to each attribute combination, and combining the reference text features corresponding to all attribute combinations to obtain the text feature set. Among them, the attribute combination corresponds to multiple attributes, and the description text includes multiple attributes of the corresponding attribute combination and the description content used to describe each attribute. Therefore, the reference text features obtained based on the description text corresponding to each attribute combination have higher adaptability to multi-attribute recognition, and the reference text features also incorporate the attribute features of the corresponding attribute combination, so as to better meet the feature requirements for multi-attribute recognition. Based on the features of the image to be identified and multiple reference text features in the text feature set, the reference text features that match the image to be identified are determined, so that the attribute combination matched by the image to be identified is determined using the reference text features that better meet the needs of multi-attribute recognition, and multiple attributes matched by the image to be identified are obtained, thereby improving the accuracy of multi-attribute recognition.

[0032] See also Figure 2 , Figure 2 It is a flow chart of another embodiment of the multi-attribute recognition method of the present application, and the multi-attribute recognition method is implemented by using a multimodal neural network model, the multimodal neural network model includes an image encoder and a text encoder, and the method includes:

[0033] S201: In response to obtaining an image to be recognized, inputting the image to be recognized into an image encoder to obtain features of the image to be recognized in a matrix form; wherein the features of the image to be recognized in a matrix form correspond to a first feature matrix.

[0034] Specifically, when an image to be identified including a target is obtained, the image to be identified is input into an image encoder for feature recognition to obtain features of the image to be identified in a matrix form, wherein the features of the image to be identified in a matrix form correspond to a first feature matrix.

[0035] It should be noted that after the image features to be identified are input into the image encoder, the image block features corresponding to multiple image blocks together form a first feature matrix, and the dimension of the first feature matrix is ​​related to the number of image blocks.

[0036] For ease of explanation, take the image encoder as Vision Transformer, with a total of 12 transformer layers as an example, input the image to be recognized into the image encoder, and the image to be recognized will be transformer encoded into the first feature matrix after being sent to the image encoder, where the dimension of the first feature matrix corresponds to the number of image blocks of the image to be recognized and the number of channels of the image encoder. The image to be recognized is divided into 14×14 image blocks, a total of 196, and all image blocks correspond to one image header, and the number of channels of the image encoder is 512. Therefore, the dimension of the first feature matrix is ​​[197, 512].

[0037] S202: Acquire a text feature set in a matrix form configured for the image to be recognized; wherein the text feature set in a matrix form corresponds to a second feature matrix.

[0038] Specifically, the text feature set includes reference text features corresponding to a plurality of attribute combinations, the attribute combinations include a plurality of attributes, and the plurality of reference text features constitute the text feature set in a matrix form.

[0039] It should be noted that the reference text features are obtained when training the multimodal neural network model. The multimodal neural network model includes an image encoder and a text encoder. The process of obtaining the reference text features includes: obtaining training sample pairs corresponding to multiple attribute combinations; wherein the training sample pairs include training images and description texts matched by the attribute combinations; inputting the training images into the image encoder to obtain training image features, inputting the description text into the text encoder to obtain training text features composed of training vectors and attribute features of attribute combinations paired together, and obtaining a predicted attribute set corresponding to each training image based on each training image feature and all training text features; adjusting the training vector based on the predicted attribute set and the attribute combination until the preset convergence condition is met, and using the trained training vector as the reference vector; pairing the reference vector with the attribute features of each attribute combination respectively to obtain the reference text features corresponding to each attribute combination.

[0040] Specifically, each attribute combination corresponds to at least one training sample pair, and the training sample pair includes a training image and a description text. The training image is input into the image encoder to obtain the training image features output by the image encoder, and the description text is input into the text encoder to obtain the training text features output by the text encoder, wherein the training text features are obtained by pairing the training vector with the attribute features of the attribute combination, and based on each training image feature and all training text features, the predicted attribute set corresponding to each training image in this training iteration is determined, wherein in each training iteration, all training texts have corresponding training text features, and the predicted attribute set includes the attribute combinations corresponding to the training text features that meet the matching conditions with the training image features.

[0041] Furthermore, based on the predicted attribute set and attribute combination corresponding to each training image, the training loss is determined, and the training vector is adjusted using the training loss until the preset convergence condition is met. The trained training vector is used as a reference vector, and the reference vector is paired with the attribute features of each attribute combination to obtain the reference text features corresponding to each attribute combination. Therefore, the reference vector is obtained after continuous adjustment based on the description text including multiple attributes during the training process, so that a reference vector that is more in line with the needs of multi-attribute recognition can be obtained, so that the reference text features can be better adapted to the multi-attribute recognition scenario.

[0042] Optionally, during the training process, a cross entropy loss is used for the learnable training vector to minimize the prediction error, while keeping the parameters of the multimodal neural network model unchanged, and the gradient can be back-propagated through the text encoder to learn task-related information. In addition, the training loss can also be determined based on other methods, and the preset convergence condition is at least related to the training loss, which is not specifically limited in this application.

[0043] It should be noted that the number of training images corresponding to each attribute combination is less than the quantity threshold, each attribute combination corresponds to at least one descriptive text in a descriptive form, and the training vector is shared among all attribute combinations during the training process. The preset convergence condition of the training vector includes the vector length of the reference vector, and the vector length is adaptively matched with the descriptive text during the training process.

[0044] Specifically, the attribute combination includes multiple attributes, and it is difficult to collect training images that include multiple attributes at the same time. The number of training images corresponding to each attribute combination is less than the quantity threshold, which can effectively reduce the difficulty of sample collection and realize multi-attribute recognition of targets based on a small number of samples. In addition, each attribute combination corresponds to at least one descriptive text in a descriptive form. The descriptive texts in multiple descriptive forms enrich the description of the image while retaining the attribute category unchanged. In the training process, the training vector is shared among all attribute combinations, so that the reference vector finally obtained can adapt to the descriptive texts in different descriptive forms of multiple attribute combinations, making the reference vector more robust.

[0045] It can be understood that the attribute combination corresponds to multiple preset attribute categories. By selecting an attribute value from each preset attribute category for combination, multiple attribute combinations can be obtained. The number of attribute combinations is positively correlated with the number of preset attribute categories and attribute values. Therefore, after the reference vector is paired with the attribute feature corresponding to each attribute combination, a multi-dimensional matrix text feature set can be obtained.

[0046] In addition, the preset convergence condition of the training vector includes the vector length of the reference vector. The training vector will be adjusted and optimized as the training process progresses, that is, the length of the training vector will be adaptively optimized and adjusted to the description text during the training process. After the training vector meets the preset convergence condition, the vector length of the obtained reference vector is adaptively matched to the description text. Therefore, the present application can customize the vector length of the reference vector based on actual conditions, thereby enhancing the adaptability of the reference vector to improve the accuracy of recognition.

[0047] For the sake of convenience, taking the multi-attribute recognition of vehicle cargo as an example, the preset attribute categories include vehicle type and cargo category. The vehicle types include 20 common vehicle types such as SUVs, commercial vehicles, sedans, pickup trucks and tractors, and the cargo categories include 10 cargo categories such as sand, wood, slag, construction waste and cartons. The two attribute categories can be combined with each other, so there are 20*10=200 attribute combinations.

[0048] It is understandable that, still taking the multi-attribute identification of vehicle cargo as an example, the preset attribute categories may also include other categories such as vehicle color and vehicle length, which are not listed in this application.

[0049] For further information, see Figure 3 , Figure 3 This is a schematic diagram of an application scenario of an implementation method of a reference text feature in this application. Taking the text encoder as BERT, with a total of 12 transformer layers as an example, when there are two preset attribute categories, the attribute features corresponding to the attribute combination include attribute value features corresponding to the two attributes, corresponding to Figure 3 In the attribute value feature 1 (the feature corresponding to the vehicle type) and the attribute value feature 2 (the feature corresponding to the cargo type), each attribute value feature corresponds to a vector representation, and the vector representations together constitute a reference vector, such as Figure 3 As shown in , the length of vector representation 1 before attribute value feature 1 is M, and the length of vector representation 2 before attribute value feature 2 is N, so the length of the reference vector is M+N. In addition, in addition to the reference vector, it also includes a text description prefix length Ls for identifying the start bit, an attribute class length Lc (the two attribute classes are Lc=2), and a text description suffix length Le for identifying the end bit. The number of attribute combinations is still 200, and the number of channels of the text image encoder is 512, for example. This application does not impose specific restrictions on this. The dimension of the second feature matrix obtained based on the above parameters is [200, Ls+M+N+Lc+Le, 512].

[0050] In a specific implementation scenario, the multiple attributes included in the attribute combination all belong to preset attribute categories. Before obtaining the text feature set in matrix form configured for the image to be recognized, it also includes: in response to the image to be recognized corresponding to the supplementary text, the supplementary text is input into the text encoder to obtain the supplementary features corresponding to the supplementary text; wherein the supplementary text includes the newly added attribute category corresponding to the image to be recognized; obtaining the text feature set in matrix form configured for the image to be recognized includes: obtaining reference text features corresponding to all attribute combinations in matrix form, fusing each reference text feature with the supplementary feature, and obtaining the text feature set in matrix form configured for the image to be recognized.

[0051] Specifically, when the image to be identified also corresponds to supplementary text, the supplementary text includes a new attribute category corresponding to the image to be identified that is different from the preset attribute category. The supplementary text is input into the text encoder to obtain supplementary features corresponding to the supplementary text, so that the supplementary features can represent the features of the new attribute category.

[0052] Furthermore, reference text features corresponding to all attribute combinations are obtained in matrix form, and each reference text feature is fused with the supplementary feature so that each reference text feature is fused with the feature of the newly added attribute category, thereby obtaining a text feature set in matrix form configured for the image to be identified, so that when the image to be identified needs to identify the newly added attribute category, the text feature set can match the actual demand, thereby improving the flexibility of multi-attribute recognition.

[0053] Optionally, when fusing the reference text feature with the supplementary feature, the reference text feature and the supplementary feature are concatenated, thereby adding the supplementary feature to the reference text feature without changing the reference text feature.

[0054] In a specific implementation scenario, each of the multiple attributes included in the attribute combination has its own preset attribute value. Before obtaining the text feature set in the form of a matrix configured for the image to be recognized, it also includes: obtaining the attribute values ​​to be adjusted corresponding to at least some of the attributes in the attribute combination, and determining the attribute adjustment features corresponding to the attribute values ​​to be adjusted; obtaining the text feature set in the form of a matrix configured for the image to be recognized, including: obtaining reference text features corresponding to all attribute combinations in the matrix form, adjusting the attribute features in each reference text feature using the attribute adjustment features, and obtaining the text feature set in the form of a matrix configured for the image to be recognized.

[0055] Specifically, when the attribute values ​​of at least some attributes in the attribute combination need to be adjusted, the attribute values ​​to be adjusted corresponding to the attributes to be adjusted in the attribute combination are obtained, and the attribute adjustment features corresponding to the attribute values ​​to be adjusted are determined. Taking vehicle type as an attribute category as an example, the attribute value is a specific value matching the attribute category such as off-road vehicle, commercial vehicle, sedan, pickup truck and tractor, and the attribute value adjustment includes at least one of adding, deleting and modifying.

[0056] Furthermore, reference text features corresponding to all attribute combinations in matrix form are obtained, and the attribute features in each reference text feature are adjusted using the attribute adjustment features, so that each reference text feature is updated to a form with the highest matching degree with the attribute value in the attribute category, and a text feature set in matrix form configured for the image to be recognized is obtained, so that when the image to be recognized needs to be recognized to obtain the adjusted attribute value, the text feature set can match the actual needs, thereby improving the flexibility of multi-attribute recognition.

[0057] It is understandable that the above-mentioned different implementation scenarios can also be used in combination to meet different practical needs, and this application will not go into details.

[0058] S203: Determine the attribute combination matched by the image to be identified based on the image features to be identified and the text feature set.

[0059] Specifically, based on the image features to be identified and multiple reference text features in the text feature set, the reference text features that match the image to be identified are determined, so that the attribute combination matched by the image to be identified is determined using the reference text features that better meet the requirements of multi-attribute recognition.

[0060] In some implementation scenarios, the first feature matrix and the second feature matrix are respectively subjected to dimensionality reduction processing to obtain a first adjustment matrix corresponding to the first feature matrix and a second adjustment matrix corresponding to the second feature matrix; based on the first adjustment matrix and the second adjustment matrix, reference text features that match the image features to be identified are determined; and at least the attribute combination included in the reference text features that match the image features to be identified is used as the attribute combination matched by the image to be identified.

[0061] Specifically, the first feature matrix and the second feature matrix are respectively subjected to dimensionality reduction processing to obtain a first adjustment matrix corresponding to the first feature matrix after the dimensionality reduction processing, and a second adjustment matrix corresponding to the second feature matrix, thereby reducing the computational complexity when the first adjustment matrix and the second adjustment matrix are matched, and the image features to be identified in the first adjustment matrix and the reference text features in the second adjustment matrix are matched to determine the reference text features that match the image features to be identified.

[0062] Furthermore, the attribute combination included in the reference text feature that matches the image feature to be identified is obtained, and it is determined whether the reference text feature matches the preset attribute category and the attribute value included therein. If it is a complete match, the attribute combination included in the reference text feature that matches the image feature to be identified is used as the attribute combination matched by the image to be identified. If it is not a complete match, all adjusted attributes in the reference text feature are determined, and all adjusted attributes included in the reference text feature that matches the image feature to be identified are used as the attribute combination matched by the image to be identified.

[0063] It is understandable that, in the process of matching the first adjustment matrix and the second adjustment matrix, the cosine similarity of the first adjustment matrix and the second adjustment matrix may be calculated, and the reference text feature with the highest similarity to the image feature to be identified may be used as the matching result.

[0064] Optionally, the dimension of the first feature matrix is ​​related to the number of image blocks corresponding to the image to be identified, the first feature matrix includes image block features corresponding to each image block, and the image block features include global features of the image to be identified, and the dimension of the second feature matrix is ​​related to the number of attribute combinations and reference text features. Performing dimensionality reduction processing on the first feature matrix and the second feature matrix respectively to obtain a first adjustment matrix corresponding to the first feature matrix and a second adjustment matrix corresponding to the second feature matrix includes: performing fixed position sampling on the first feature matrix and the second feature matrix respectively and converting them to the same space to obtain a first adjustment matrix corresponding to the first feature matrix and a second adjustment matrix corresponding to the second feature matrix.

[0065] Specifically, both the image encoder and the text encoder include multiple transformer layers. Combining the specific dimensions of the first feature matrix and the second feature matrix, it can be seen that after the image features to be identified are input into the image encoder, the image block features corresponding to multiple image blocks together form the first feature matrix. The dimension of the first feature matrix is ​​related to the number of image blocks. After being processed by the transformer layer, each image block feature includes the global features of the image to be identified. In addition, the dimension of the second feature matrix is ​​related to the number of attribute combinations and the length of the reference vector included in the reference text features.

[0066] Furthermore, the first characteristic matrix and the second characteristic matrix are sampled at fixed positions respectively to reduce the matrix dimension and maintain the consistency of matrix processing, and the sampled first characteristic matrix and the second characteristic matrix are projected to convert the matrices to the same space to obtain a first adjustment matrix corresponding to the first characteristic matrix and a second adjustment matrix corresponding to the second characteristic matrix, so as to effectively reduce the complexity of matrix matching and ensure the consistency of matrix processing.

[0067] In this embodiment, when an image to be identified including a target is obtained, the image to be identified is input into an image encoder for feature identification, and features of the image to be identified in a matrix form are obtained. The text feature set includes reference text features corresponding to multiple attribute combinations, and the reference text features are obtained by pairing reference vectors and attribute features. The reference text features are obtained after continuous adjustment based on the description text of multiple attributes during the training process, so that a reference vector that is more in line with the needs of multi-attribute identification can be obtained. After the reference vector is paired with the attribute features corresponding to each attribute combination, a multi-dimensional text feature set in a matrix form can be obtained. The first feature matrix and the second feature matrix are respectively subjected to dimensionality reduction processing to obtain a first adjustment matrix corresponding to the first feature matrix after dimensionality reduction processing, and a second adjustment matrix corresponding to the second feature matrix, so as to reduce the computational complexity of matching the first adjustment matrix and the second adjustment matrix, match the image features to be identified in the first adjustment matrix and the reference text features in the second adjustment matrix, determine the reference text features that match the image features to be identified, and at least use the attribute combination included in the reference text features that match the image features to be identified as the attribute combination matched by the image to be identified, so as to improve the accuracy of multi-attribute identification.

[0068] See also Figure 4 , Figure 4 It is a structural diagram of an embodiment of an electronic device of the present application, wherein the electronic device 40 includes a memory 401 and a processor 402 coupled to each other, wherein the memory 401 stores program data (not shown), and the processor 402 calls the program data to implement the method in any of the above embodiments. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0069] See also Figure 5 , Figure 5 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 50 stores program data 500. When the program data 500 is executed by a processor, the method in any of the above embodiments is implemented. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0070] It should be noted that the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present implementation scheme.

[0071] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0072] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0073] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multi-attribute recognition method, characterized in that: The method comprises: Acquire the image features to be recognized corresponding to the image to be recognized, and the text feature set configured for the image to be recognized; wherein the text feature set is configured based on the following steps, including: acquiring description texts corresponding to each of a plurality of attribute combinations, and the attribute features corresponding to each of the attribute combinations; based on the description text corresponding to each of the attribute combinations and the attribute features corresponding to each of the attribute combinations, determining the reference text features corresponding to each of the attribute combinations, and obtaining the text feature set composed of the reference text features corresponding to each of the attribute combinations; the description text includes the multiple attributes included in the attribute combination and the description content corresponding to each of the attributes; Based on the features of the image to be recognized and the text feature set, a combination of attributes matched by the image to be recognized is determined.

2. The method according to claim 1, characterized in that The reference text features are obtained when training a multimodal neural network model, wherein the multimodal neural network model includes an image encoder and a text encoder. The process of obtaining the reference text features includes: Acquire a plurality of training sample pairs corresponding to each of the attribute combinations; wherein the training sample pairs include a training image matched by the attribute combination and the description text; Input the training image into the image encoder to obtain training image features, input the description text into the text encoder to obtain training text features composed of training vectors and attribute features of attribute combinations, and obtain a prediction attribute set corresponding to each training image based on each training image feature and all training text features; Based on the prediction attribute set and the attribute combination, the training vector is adjusted until a preset convergence condition is satisfied, and the trained training vector is used as a reference vector; The reference vector is paired with the attribute feature of each attribute combination respectively to obtain the reference text feature corresponding to each attribute combination.

3. The method according to claim 2, characterized in that The number of training images corresponding to each of the attribute combinations is less than a quantity threshold, each of the attribute combinations corresponds to at least one descriptive form of the description text, and during the training process, the training vector is shared among all of the attribute combinations, and the preset convergence condition of the training vector includes the vector length of the reference vector, and the vector length is adaptively matched with the description text during the training process.

4. The method according to claim 1, characterized in that: The method is implemented by using a multimodal neural network model, which includes an image encoder and a text encoder; The step of obtaining the image features to be recognized corresponding to the image to be recognized and the text feature set configured for the image to be recognized includes: In response to obtaining the image to be recognized, inputting the image to be recognized into the image encoder to obtain features of the image to be recognized in a matrix form; wherein the features of the image to be recognized in a matrix form correspond to a first feature matrix; A text feature set in matrix form configured for the image to be recognized is obtained; wherein the text feature set in matrix form corresponds to a second feature matrix.

5. The method according to claim 4, characterized in that The multiple attributes included in the attribute combination all belong to the preset attribute category; Before acquiring the text feature set in the form of a matrix configured for the image to be recognized, the method further includes: In response to the image to be identified corresponding to a supplementary text, the supplementary text is input into the text encoder to obtain a supplementary feature corresponding to the supplementary text; wherein the supplementary text includes a newly added attribute category corresponding to the image to be identified; The step of obtaining a text feature set in a matrix form configured for the image to be recognized includes: Reference text features corresponding to all the attribute combinations in matrix form are obtained, and each of the reference text features is fused with the supplementary feature to obtain a text feature set in matrix form configured for the image to be recognized.

6. The method according to claim 4, characterized in that Among the multiple attributes included in the attribute combination, each attribute corresponds to its own preset attribute value; Before acquiring the text feature set in the form of a matrix configured for the image to be recognized, the method further includes: Acquire attribute values ​​to be adjusted corresponding to at least some attributes in the attribute combination, and determine attribute adjustment features corresponding to the attribute values ​​to be adjusted; The step of obtaining a text feature set in a matrix form configured for the image to be recognized includes: Reference text features corresponding to all attribute combinations in matrix form are obtained, and the attribute feature in each of the reference text features is adjusted using the attribute adjustment feature to obtain a text feature set in matrix form configured for the image to be recognized.

7. The method according to claim 4, characterized in that The step of determining the attribute combination matched by the image to be identified based on the image feature to be identified and the text feature set includes: Performing dimensionality reduction processing on the first characteristic matrix and the second characteristic matrix respectively to obtain a first adjustment matrix corresponding to the first characteristic matrix and a second adjustment matrix corresponding to the second characteristic matrix; Determining reference text features that match the image features to be recognized based on the first adjustment matrix and the second adjustment matrix; At least the attribute combination included in the reference text feature that matches the image feature to be recognized is used as the attribute combination matched by the image to be recognized.

8. The method according to claim 7, characterized in that The dimension of the first feature matrix is ​​related to the number of image blocks corresponding to the image to be identified, the first feature matrix includes image block features corresponding to each image block, and the image block features include global features of the image to be identified, and the dimension of the second feature matrix is ​​related to the number of attribute combinations and the reference text features; The performing dimensionality reduction processing on the first characteristic matrix and the second characteristic matrix respectively to obtain a first adjustment matrix corresponding to the first characteristic matrix and a second adjustment matrix corresponding to the second characteristic matrix includes: The first feature matrix and the second feature matrix are respectively sampled at fixed positions and converted to the same space to obtain a first adjustment matrix corresponding to the first feature matrix and a second adjustment matrix corresponding to the second feature matrix.

9. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Image recognition method, device and system and storage medium

    CN108875932A

  • Method, system and application for intelligently identifying and analyzing implicit attributes of roles in digital reading

    CN116204642A

  • Attribute-learning and -transferring system, recognizer generation device, recognizer generation method and recognition device

    JP2012084117A

  • Identifying digital attributes from multiple attribute groups within target digital images utilizing a deep cognitive attribution neural network

    US20210073267A1