Image feature extraction method and device, storage medium and terminal

Abstract

The present invention discloses an image feature extraction method and device, a storage medium, and a terminal. The method utilizes a trained image feature extraction model. The image feature extraction model includes a visual region feature extraction unit for performing preliminary feature extraction on image data to obtain region relationship attribute features and grid block local features; an attribute space information extraction unit for performing attribute space information extraction on grid block local features to obtain an attribute space embedding vector; an attribute semantic information extraction unit for performing attribute semantic information extraction on text data to obtain an attribute semantic embedding vector; and an attribute relationship extraction module for modeling attribute relationships based on the attribute space embedding vector and the attribute semantic embedding vector to obtain attribute relationship attribute features that model spatial dimension attribute relationships and semantic dimension attribute relationships. The present invention solves the problem of low object attribute recognition accuracy caused by insufficient modeling of relationships in images by feature extraction methods in object attribute recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to an image feature extraction method and device, a storage medium and a terminal. Background Art

[0002] Object attribute recognition refers to identifying a series of real properties contained in an object within an image. For example, a face in a face image contains attributes such as age, gender, and whether it wears glasses, and a pedestrian in a pedestrian image contains attributes such as whether it wears a hat, the color of its upper body clothing, and the color of its lower body clothing. Object attribute recognition can be widely applied in different fields. For example, pedestrian attribute recognition (also known as pedestrian attribute recognition) in pedestrian images can provide additional attribute basis for pedestrian re-identification algorithms to determine pedestrian categories, and is widely used in the field of video surveillance. Feature extraction is a key step in object attribute recognition. Extracting attribute features with strong discriminative capabilities can effectively improve the accuracy of object attribute recognition. Therefore, studying feature extraction in object attribute recognition is of great significance.

[0003] Commonly used feature extraction methods for object attribute recognition fall into three categories: global feature-based, local feature-based, and relation-based. Global feature-based methods extract a single global feature representation for all attributes, using this single global feature to identify all attributes. Local feature-based methods first locate the attribute position, then extract attribute features based on this positional information, and finally use these features to identify the attribute. These two methods are relatively early research approaches and overlook the importance of relationships in object attribute recognition. Object attribute recognition requires regional relationships. For example, when identifying the pedestrian attribute "age" in pedestrian images, humans focus on the head, upper body, and lower body regions, or consider all three regions together. Inspired by this phenomenon, relation-based feature extraction methods attempt to establish regional relationships within the image while extracting attribute features. Object attribute recognition requires attribute relationships, and attributes generally have strong correlations. For example, in pedestrian images, the pedestrian attribute "female" often appears together with the attribute "dress." The presence or absence of the attribute "dress" significantly affects the probability of the attribute "female" appearing. Inspired by the above phenomenon, since attributes have information in spatial and semantic dimensions, the relationship-based feature extraction method will try to establish attribute relationships in these two dimensions while extracting attribute features.

[0004] Current methods do not adequately model relationships when extracting attribute features, and often neglect the modeling of regional relationships or spatial dimension attribute relationships, resulting in weak attribute feature discrimination capabilities and low object attribute recognition accuracy. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that the existing image feature extraction methods do not adequately model the relationships when extracting attribute features, and often ignore the modeling of regional relationships or spatial dimension attribute relationships, resulting in weak attribute feature discrimination ability and low object attribute recognition accuracy.

[0006] In order to solve the above technical problems, the present invention provides an image feature extraction method, comprising:

[0007] Acquire an object attribute recognition data set, wherein the object attribute recognition data set includes a plurality of object attribute recognition data, and the object attribute recognition data includes image data and text description data of target object attributes in the image data;

[0008] Performing feature extraction on each object attribute recognition data in the object attribute recognition data set based on the trained image feature extraction model to obtain region relationship attribute features and attribute relationship attribute features of the target object in the image data of each object attribute recognition data;

[0009] The image feature extraction model includes a visual area feature extraction unit, an attribute space information extraction unit, an attribute semantic information extraction unit and an attribute relationship extraction unit;

[0010] The visual area feature extraction unit is used to perform preliminary feature extraction on the image data to obtain regional relationship attribute features and grid block local features;

[0011] The attribute space information extraction unit is used to extract attribute space information from the local features of the grid block to obtain an attribute space embedding vector;

[0012] The attribute semantic information extraction unit is used to extract attribute semantic information from the text description data to obtain an attribute semantic embedding vector;

[0013] The attribute relationship extraction module is used to model attribute relationships based on the attribute space embedding vector and the attribute semantic embedding vector to obtain attribute relationship attribute features that model the spatial dimension attribute relationship and the semantic dimension attribute relationship.

[0014] Preferably, the visual area feature extraction unit performs preliminary feature extraction on the image data to obtain regional relationship attribute features and grid block local features, including:

[0015] Dividing the image data into a sequence of overlapping square grid blocks using a sliding window, and mapping the sequence of square grid blocks into an initial grid block embedding vector using a trainable linear mapping matrix;

[0016] Adding a learnable global embedding vector before the first position of the initial grid block embedding vector to obtain a temporary grid block embedding vector;

[0017] Adding a previous learnable grid block position embedding vector to the temporary grid block embedding vector to obtain a grid block embedding vector;

[0018] Image region relationship modeling is performed on all element relationships in the grid block embedding vector using a preset modeling tool to obtain region relationship attribute features and grid block local features.

[0019] Preferably, the attribute semantic information extraction unit extracts attribute semantic information from the text description data to obtain an attribute semantic embedding vector, including:

[0020] Extracting word embedding vectors of different attributes in the text description data through a pre-trained word vector model to obtain attribute word embedding vectors;

[0021] The attribute word embedding vector is mapped to the image domain using a learnable linear mapping matrix to obtain an attribute semantic embedding vector.

[0022] Preferably, the attribute relationship extraction module models the attribute relationship based on the attribute space embedding vector and the attribute semantic embedding vector to obtain the attribute relationship attribute features that model the spatial dimension attribute relationship and the semantic dimension attribute relationship, including:

[0023] fusing the attribute space embedding vector and the attribute semantic embedding vector to obtain a fusion result, and adding a learnable attribute position embedding vector to the fusion result to obtain an attribute space semantic embedding vector;

[0024] Attribute relationship modeling in spatial dimension and semantic dimension is performed on all element relationships in the attribute space semantic embedding vector by a preset modeling tool to obtain attribute relationship attribute features.

[0025] Preferably, the process of training the image feature extraction model includes:

[0026] Acquire an object attribute recognition training data set, the object attribute recognition training data set including a plurality of object attribute recognition training data, the object attribute recognition training data including image data, text description data of target object attributes in the image data, and label true values ​​of each attribute of the target object in the image data;

[0027] Dividing all object attribute recognition training data in the object attribute recognition training data set into multiple groups to obtain multiple groups of object attribute recognition training data groups;

[0028] Training a temporary image feature extraction model based on the Mth group of object attribute recognition training data to obtain a parameter-updated image feature extraction model, determining whether M is equal to a preset threshold, and if so, determining whether the current parameter-updated image feature extraction model meets the loss value requirements to reach a convergence state, and if so, using the current parameter-updated image feature extraction model as the trained image feature extraction model; if M is not equal to the preset threshold, adding 1 to M, using the current parameter-updated image feature extraction model as the temporary image feature extraction model, and re-training the temporary image feature extraction model based on the Mth group of object attribute recognition training data; if the current parameter-updated image feature extraction model does not meet the loss value requirements to reach a convergence state, initializing M to 1, using the current parameter-updated image feature extraction model as the temporary image feature extraction model, and re-training the temporary image feature extraction model based on the Mth group of object attribute recognition training data;

[0029] Among them, the preset threshold is the total number of object attribute recognition training data groups in the object attribute recognition training data set, and the initial value of M is 1; when training is performed for the first time based on the object attribute recognition training data group, the temporary image feature extraction model is the initially created image feature extraction model.

[0030] Preferably, in the process of training the temporary image feature extraction model based on the object attribute recognition training data set, the process of optimizing the parameters of the temporary image feature model includes:

[0031] inputting the region relationship attribute feature and the attribute relationship attribute feature of each object attribute recognition training data in the object attribute recognition training data group into the attribute classifier respectively to obtain a first attribute prediction score and a second attribute prediction score of each object attribute recognition training data;

[0032] Calculating a first binary cross entropy loss value corresponding to the first attribute prediction score of each object attribute recognition training data based on the first attribute prediction score of each object attribute recognition training data, and calculating a second binary cross entropy loss value corresponding to the second attribute prediction score of each object attribute recognition training data based on the second attribute prediction score of each object attribute recognition training data;

[0033] Obtaining a loss value of a feature extraction model loss function based on a first binary cross entropy loss value of a first attribute prediction score and a second binary cross entropy loss value of a second attribute prediction score of all the object attribute recognition training data;

[0034] The temporary image feature model parameters are optimized based on the loss value of the feature extraction model loss function.

[0035] Preferably, the expression of the binary cross entropy loss value is:

[0036]

[0037] Among them, L i The i-th binary cross entropy loss value represents the prediction score of the i-th attribute of the object attribute recognition training data, i∈(1,2), y i c represents the i-th predicted score value of the attribute classifier on the c-th attribute, Represents the true value of the label of the cth attribute of the input image data, ρ c =exp(-r c ) represents the weight of the cth attribute, r c represents the proportion of the cth attribute in all samples in the object attribute recognition training dataset, and σ(·) represents the sigmoid activation function;

[0038] The feature extraction model loss function is:

[0039] L=L1+λL2

[0040] Among them, λ represents the weight parameter.

[0041] In order to solve the above technical problems, the present invention also provides an image feature extraction device, comprising a data acquisition module and a feature extraction module;

[0042] The data acquisition module is used to acquire an object attribute recognition data set, wherein the object attribute recognition data set includes a plurality of object attribute recognition data, and the object attribute recognition data includes image data and text description data of target object attributes in the image data;

[0043] The feature extraction module is used to extract features from each object attribute recognition data in the object attribute recognition data set based on the trained image feature extraction model to obtain the region relationship attribute features and attribute relationship attribute features of the target object in the image data of each object attribute recognition data;

[0044] The image feature extraction model includes a visual area feature extraction unit, an attribute space information extraction unit, an attribute semantic information extraction unit and an attribute relationship extraction unit;

[0045] The visual area feature extraction unit is used to perform preliminary feature extraction on the image data to obtain regional relationship attribute features and grid block local features;

[0046] The attribute space information extraction unit is used to extract attribute space information from the local features of the grid block to obtain an attribute space embedding vector;

[0047] The attribute semantic information extraction unit is used to extract attribute semantic information from the text description data to obtain an attribute semantic embedding vector;

[0048] The attribute relationship extraction module is used to model attribute relationships based on the attribute space embedding vector and the attribute semantic embedding vector to obtain attribute relationship attribute features that model the spatial dimension attribute relationship and the semantic dimension attribute relationship.

[0049] In order to solve the above technical problems, the present invention further provides a storage medium on which a computer program is stored, and when the program is executed by a processor, an image feature extraction method is implemented.

[0050] In order to solve the above technical problem, the present invention further provides a terminal, characterized by comprising: a processor and a memory, wherein the memory is communicatively connected to the processor;

[0051] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal performs an image feature extraction method.

[0052] Compared with the prior art, one or more embodiments of the above solutions may have the following advantages or beneficial effects:

[0053] The image feature extraction method provided by the embodiment of the present invention provides an image feature extraction method based on regional relationships and attribute relationships, which solves the problem of low object attribute recognition accuracy caused by insufficient modeling of relationships in the image by feature extraction methods in object attribute recognition. Specifically, the method of the present invention implements regional relationship modeling through a visual region feature extraction unit, and designs an attribute relationship module to implement attribute relationship modeling in spatial and semantic dimensions. It fully explores the regional relationships and attribute relationships in the image, extracts attribute features that contain rich relationship information, enhances the ability to distinguish attributes, and improves the accuracy of object attribute recognition.

[0054] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0056] Figure 1 A schematic diagram showing the flow of an image feature extraction method according to a first embodiment of the present invention is shown;

[0057] Figure 2 A schematic diagram of a model framework of an image feature extraction model in the first embodiment of the present invention is shown;

[0058] Figure 3 A schematic diagram of the structure of a multi-layer Transformer encoder in embodiment 1 of the present invention is shown;

[0059] Figure 4 A schematic structural diagram of an attribute space information extraction unit in the first embodiment of the present invention is shown;

[0060] Figure 5 A schematic diagram showing the structure of an attribute semantic information extraction unit in the first embodiment of the present invention is shown;

[0061] Figure 6 A schematic diagram of the structure of an image feature extraction device according to a second embodiment of the present invention is shown;

[0062] Figure 7 It shows a schematic structural diagram of a terminal according to a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings and examples, so that the present invention can fully understand how to apply technical means to solve technical problems and achieve technical effects, and thus implement the invention accordingly. It should be noted that, as long as no conflict exists, the various embodiments of the present invention and the various features of the embodiments can be combined with each other, and the resulting technical solutions are all within the scope of protection of the present invention.

[0064] Current image feature extraction methods fail to adequately model relationships when extracting attribute features, often neglecting to model regional or spatial attribute relationships. This results in weak attribute feature discrimination and low object attribute recognition accuracy. Therefore, establishing attribute relationships across regions, spatial dimensions, and semantic dimensions during attribute feature extraction is key to further improving object attribute recognition accuracy.

[0065] Example 1

[0066] In order to solve the technical problems existing in the prior art, an embodiment of the present invention provides an image feature extraction method.

[0067] Figure 1 FIG. 1 shows a flow chart of an image feature extraction method according to an embodiment of the present invention; FIG. Figure 1 As shown, the image feature extraction method according to the embodiment of the present invention includes the following steps.

[0068] Step S101: Acquire an object attribute recognition dataset.

[0069] Specifically, the object attribute recognition dataset contains multiple object attribute recognition data. Each object attribute recognition data includes image data and text description data of the target object attributes in the image data. For example, the target object can be set as a pedestrian, the corresponding image data can be set as pedestrian image data, and the text description data can be set as pedestrian attribute text description data. The pedestrian attributes in the pedestrian image data may include whether the pedestrian is wearing a hat, the color of the upper body clothing, the color of the lower body clothing, and other attributes.

[0070] Step S102 : performing feature extraction on each object attribute recognition data in the object attribute recognition data set based on the trained image feature extraction model to obtain the region relationship attribute feature and attribute relationship attribute feature of the image data target object in each object attribute recognition data.

[0071] Specifically, all object attribute recognition data in the object attribute recognition dataset are input into the trained image feature extraction model in sequence to obtain the region relationship attribute features and attribute relationship attribute features of the target object in each object attribute recognition data in the object attribute recognition dataset.

[0072] in Figure 2 FIG1 shows a schematic diagram of a model framework of an image feature extraction model in the first embodiment of the present invention; ... Figure 2 As shown, the acquisition of the trained image feature extraction model includes the following steps:

[0073] Step S201: construct an image feature extraction model.

[0074] Specifically, the constructed image feature extraction model includes a visual area feature extraction unit (ie, a visual Transformer feature extraction unit), an attribute space information extraction unit, an attribute semantic information extraction unit, and an attribute relationship extraction unit.

[0075] The visual region feature extraction unit is mainly used to perform preliminary feature extraction on the image data to obtain the regional relationship attribute features and grid block local features that model the regional relationship. Furthermore, the image data is first divided into a sequence of overlapping square grid blocks using a sliding window. Assume that the input image data size is H×W, the step size of the sliding window is S, the side length of the square grid block is P, and the input image data is divided into N grid blocks, then N satisfies the following formula: Where [·] represents a rounding-down operation, and the overlapping area of ​​a single square grid block is P×(PS).

[0076] Then, the trainable linear mapping matrix E is used to transform the square grid block dimension from (3×p 2 ) dimension is mapped to D dimension to output the initial grid block embedding vector

[0077] Then a learnable global embedding vector X class , placed in front of the initial grid block embedding vector to obtain a temporary grid block embedding vector, which is expressed as [X class ,X 1 E,X 2 E,…,X N E].

[0078] In order to capture the position information between grid blocks, a learnable grid block position embedding vector is added to the temporary grid block embedding vector To obtain the final form of the grid block embedding vector In order to model the image region relationship, the final form of the grid block embedding vector z0 is input into a preset modeling tool to obtain the region relationship attribute features and grid block local features. Preferably, the preset modeling tool is a multi-layer Transformer encoder. Figure 3 A schematic diagram of the structure of a multi-layer Transformer encoder in the first embodiment of the present invention is shown; Figure 3 As shown, the grid block embedding vector z0 is input into the Transformer encoder containing L layers. The output of the lth layer Transformer Encoder layer (Transformer EncoderLayer) is represented as:

[0079] Z' l =MSA(LN(Z l-1 ))+Z l-1 , l=1,…,L

[0080] Z l =MLP(LN(Z' l ))+Z' l , l=1,……,L

[0081] Among them, Z l denotes the output grid block embedding vector of the l-th Transformer encoding layer, MSA denotes the Multi-head Self-attention module (MSA), MLP denotes the Multilayer Perceptron (MLP), LN denotes the LayerNorm (LN), and l denotes the l-th layer of the multi-layer Transformer encoder.

[0082] Then the first element of the last layer output of the multi-layer Transformer encoder (i.e. ) Copy C copies, that is, obtain the regional relationship attribute characteristics Where C represents the number of attributes of the target object,

[0083] Merge the last N elements of the last layer output of the multi-layer Transformer encoder to form the local features of the grid block in

[0084] It should be noted that, whether in the training process or the actual feature extraction process, the visual area feature extraction unit obtains the regional relationship attribute features and grid block local features of the image data in the above manner.

[0085] The attribute space information extraction unit is mainly used to extract attribute space information from local features of grid blocks to obtain attribute space embedding vectors. Figure 4 The structure diagram of the attribute space information extraction unit in the first embodiment of the present invention is shown; Figure 4 As shown, further, first based on the local features of the grid block Get the attribute attention value V of the spatial dimension. The attribute attention value V is expressed as:

[0086]

[0087] Among them, FC represents a single fully connected layer, which is used to map the dimension from D to C, and V represents the unregularized attribute attention value. Then the Softmax loss function is applied to the attribute attention value V to obtain the attention map A, which is expressed as:

[0088]

[0089] Among them, A c,i Indicates the degree of attention of the cth attribute to the i-th region. The embedding vector of the cth attribute in the spatial dimension is equal to the attention weight A of all elements in the input sequence c The weighted average of is expressed as:

[0090]

[0091] Among them, SpE represents the attribute space embedding vector.

[0092] It should be noted that, whether in the training process or the actual feature extraction process, the attribute space information extraction unit obtains the attribute space embedding vector in the above manner.

[0093] The attribute semantic information extraction unit is used to extract attribute semantic information from text data to obtain an attribute semantic embedding vector. Figure 5 The structure diagram of the attribute semantic information extraction unit in the first embodiment of the present invention is shown; Figure 5 As shown, further, we first need to extract the word embedding vectors of different attributes in the text data through the pre-trained word vector model to obtain the attribute word embedding vector The pre-trained word vector model can be the bert-base-uncased model. Then, using a learnable linear mapping matrix, the attribute word embedding vector of the text domain is mapped to the image domain to obtain the attribute semantic embedding vector SeE, which is expressed as:

[0094]

[0095] in, Represents a learnable linear mapping matrix.

[0096] It should be noted that, whether in the training process or the actual feature extraction process, the attribute semantic information extraction unit obtains the attribute semantic embedding vector in the above manner.

[0097] The attribute relationship extraction module is used to model attribute relationships based on attribute spatial embedding vectors and attribute semantic embedding vectors to obtain attribute relationship attribute features that model attribute relationships in spatial dimensions and semantic dimensions. Furthermore, the attribute spatial embedding vector SpE and the attribute semantic embedding vector SeE must first be fused. The fusion process can be the sum of the two vectors to obtain the fusion result. Then, in order to capture the position information between attributes, the fusion result is added with a learnable attribute position embedding vector To obtain the attribute space semantic embedding vector, the attribute space semantic embedding vector can be expressed as: SpeE=SpE+SeE+Spos.

[0098] In order to model the attribute relationship between the spatial dimension and the semantic dimension, the attribute space semantic embedding vector SpeE is input into the preset tool to obtain the attribute relationship attribute features that model the attribute relationship between the spatial dimension and the semantic dimension; further, referring to Figure 3 As shown in Figure 2, this process is implemented based on a multi-layer Transformer encoder. The output of the preset tool is the attribute relationship attribute feature F that models the relationship between spatial and semantic dimensions. r =[F r 1 ,F r 2 ,F r 3 ,……,F r C ].

[0099] It should be noted that, whether in the training process or the actual feature extraction process, the attribute relationship extraction module obtains the attribute relationship attribute features that model the attribute relationship between the spatial dimension and the semantic dimension through the above method.

[0100] Step S202: training the image feature extraction model.

[0101] Specifically, we first need to obtain an object attribute recognition training data set. Similar to the object attribute recognition data set, the object attribute recognition training data set also contains multiple object attribute recognition training data. Each object attribute recognition training data contains image data and text description data of the target object attributes in the image data. At the same time, the object attribute recognition training data in the object attribute recognition training data set also includes the label true value data of each attribute of the target object in the image data.

[0102] The image feature extraction model is then trained based on the object attribute recognition training dataset to obtain a trained image feature extraction model. Furthermore, all object attribute recognition training data in the object attribute recognition training dataset is divided into multiple groups to obtain multiple object attribute recognition training data groups. A temporary image feature extraction model is then trained based on the Mth group of object attribute recognition training data groups, and during training, various parameters in the temporary image feature extraction model are adjusted based on the loss value calculated by the loss function to obtain a parameter-updated image feature extraction model. A determination is then made as to whether M is equal to a preset threshold. If so, a determination is made as to whether the current parameter-updated image feature extraction model meets the loss value convergence state. If so, the current parameter-updated image feature extraction model is used as the trained image feature extraction model. If M is not equal to the preset threshold, M is incremented by 1, and the temporary image feature extraction model is retrained based on the Mth group of object attribute recognition training data groups. If the current parameter-updated image feature extraction model does not meet the loss value convergence state, M is initialized to 1, and the temporary image feature extraction model is retrained based on the Mth group of object attribute recognition training data groups.

[0103] It should be noted that the preset threshold is the total number of object attribute recognition training data groups in the object attribute recognition training data set, and the initial value of M is 1; when training is performed for the first time based on the object attribute recognition training data group, the temporary image feature extraction model is the initially created image feature extraction model.

[0104] Since each parameter in the image feature extraction model needs to be adjusted based on the loss value calculated by the loss function during each training, the training process based on multiple groups of object attribute recognition training data groups realizes the process of multiple optimization of parameters in the image feature extraction model; repeat the above process of training the image feature extraction model based on each object attribute recognition training data group separately for multiple rounds until the loss value no longer continues to decrease and reaches a convergence state, that is, the currently obtained parameter-updated image feature extraction model has reached a convergence state in accordance with the loss value, and the trained image feature extraction model can be obtained, and the trained image feature extraction model is used as the trained image feature extraction model.

[0105] It should be noted that each unit in the trained image feature extraction model is obtained by repeatedly adjusting various parameters after training each unit with the object attribute recognition data training dataset. Therefore, when describing the working principles of each unit in the image feature extraction model above, we also described the working principles of each unit in the trained image feature extraction model.

[0106] In the actual image feature extraction process, the image feature extraction model only needs to extract the region relationship attribute features and attribute relationship attribute features of the target object in each object attribute recognition data set. However, when training the temporary image feature extraction model based on the object attribute recognition training data set, the optimization process of the temporary image feature model parameters includes:

[0107] The region relationship attribute features and attribute relationship attribute features of each object attribute recognition training data in the object attribute recognition training data set are respectively input into the attribute classifier to obtain a first attribute prediction score and a second attribute prediction score for each object attribute recognition training data. A first binary cross-entropy loss value for the first attribute prediction score of each object attribute recognition training data is then calculated based on the first attribute prediction score of each object attribute recognition training data, and a second binary cross-entropy loss value for the second attribute prediction score of each object attribute recognition training data is calculated based on the second attribute prediction score of each object attribute recognition training data. A loss value of the feature extraction model loss function is then obtained based on the first binary cross-entropy loss value and the second binary cross-entropy loss value of the first attribute prediction score and the second binary cross-entropy loss value of the second attribute prediction score of each object attribute recognition training data. A loss function loss value can be obtained based on the first binary cross-entropy loss value and the second binary cross-entropy loss value of the first attribute prediction score and the second attribute prediction score of each object attribute recognition training data. The loss value of the model loss function can be selected as the average of the loss function loss values ​​corresponding to all object attribute recognition training data in the current object attribute recognition training data set. Finally, the parameters of the temporary image feature model are optimized based on the loss value of the feature extraction model loss function.

[0108] Specifically, the processing method for the single object attribute recognition training data is as follows: each element in the regional relationship attribute feature of the object attribute recognition training data is input into the attribute classifier to obtain the first attribute prediction score of the object attribute recognition training data. Then, each element in the attribute relationship attribute feature of the object attribute recognition training data is input into the attribute classifier to obtain the second attribute prediction score of the object attribute recognition training data. Assuming that the object attribute recognition training data is the i-th data in the object attribute recognition training data group, the binary cross entropy loss expression of the first prediction score or the second prediction score of the object attribute recognition training data is:

[0109]

[0110] Among them, L i The i-th binary cross entropy loss value represents the prediction score of the i-th attribute of the object attribute recognition training data, i∈(1,2), y i c represents the i-th predicted score value of the attribute classifier on the c-th attribute, Represents the true value of the label of the cth attribute of the input image data, ρ c =exp(-r c ) represents the weight of the cth attribute, r c represents the proportion of the cth attribute in all samples in the object attribute recognition training dataset, and σ(·) represents the sigmoid activation function.

[0111] The loss function expression of the feature extraction model is:

[0112] L=L1+λL2

[0113] Among them, λ represents the weight parameter, and the value can be selected as 0.2.

[0114] The image feature extraction method provided by the embodiment of the present invention provides an image feature extraction method based on regional relationships and attribute relationships, which solves the problem of low object attribute recognition accuracy caused by insufficient modeling of relationships in the image by the feature extraction method in object attribute recognition. Specifically, the method of the present invention implements regional relationship modeling through a visual region feature extraction unit, and designs an attribute relationship module to implement attribute relationship modeling in spatial dimensions and semantic dimensions. It fully explores the regional relationships and attribute relationships in the image, extracts attribute features that contain rich relationship information, enhances the attribute discrimination ability, and improves the object attribute recognition accuracy.

[0115] Example 2

[0116] In order to solve the technical problems existing in the prior art, an embodiment of the present invention provides an image feature extraction device.

[0117] Figure 6 FIG2 shows a schematic diagram of the structure of an image feature extraction device according to the second embodiment of the present invention; Figure 6 As shown, the image feature extraction device according to the embodiment of the present invention includes a data acquisition module and a feature extraction module.

[0118] The data acquisition module is used to acquire an object attribute recognition data set, the object attribute recognition data set includes a plurality of object attribute recognition data, and the object attribute recognition data includes image data and text description data of target object attributes in the image data;

[0119] The feature extraction module is used to extract features from each object attribute recognition data in the object attribute recognition data set based on the trained image feature extraction model to obtain the region relationship attribute features and attribute relationship attribute features of the target object in the image data of each object attribute recognition data;

[0120] The image feature extraction model includes a visual area feature extraction unit, an attribute space information extraction unit, an attribute semantic information extraction unit and an attribute relationship extraction unit;

[0121] The visual region feature extraction unit is used to perform preliminary feature extraction on the image data to obtain the region relationship attribute features and grid block local features;

[0122] The attribute space information extraction unit is used to extract attribute space information from the local features of the grid block to obtain an attribute space embedding vector;

[0123] The attribute semantic information extraction unit is used to extract attribute semantic information from text data to obtain an attribute semantic embedding vector;

[0124] The attribute relationship extraction module is used to model attribute relationships based on attribute space embedding vectors and attribute semantic embedding vectors to obtain attribute relationship attribute features that model attribute relationships in spatial dimensions and attribute relationships in semantic dimensions.

[0125] The image feature extraction device provided by an embodiment of the present invention provides an image feature extraction device based on regional relationships and attribute relationships, which solves the problem of low object attribute recognition accuracy caused by insufficient modeling of relationships in images by feature extraction methods in object attribute recognition. Specifically, the device of the present invention implements regional relationship modeling through a visual region feature extraction unit, and designs an attribute relationship module to implement attribute relationship modeling in spatial and semantic dimensions. It fully explores the regional relationships and attribute relationships existing in the image, extracts attribute features that contain rich relationship information, enhances the ability to distinguish attributes, and improves the accuracy of object attribute recognition.

[0126] Example 3

[0127] In order to solve the above technical problems existing in the prior art, an embodiment of the present invention further provides a storage medium storing a computer program, which can implement all the steps in the image feature extraction method in embodiment 1 when executed by a processor.

[0128] The specific steps of the image feature extraction method and the beneficial effects obtained by applying the readable storage medium provided by the embodiment of the present invention are the same as those in the first embodiment and will not be described in detail here.

[0129] It should be noted that the storage medium includes various media that can store program codes, such as ROM, RAM, magnetic disk or optical disk.

[0130] Example 4

[0131] In order to solve the above technical problems existing in the prior art, an embodiment of the present invention further provides a terminal.

[0132] Figure 7 The schematic diagram of the terminal structure of the fourth embodiment of the present invention is shown. Figure 7 In this embodiment, the terminal includes a processor and a memory connected to each other; the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that all steps of the image feature extraction method in embodiment 1 can be implemented when the terminal executes the computer programs.

[0133] The specific steps of the image feature extraction method and the beneficial effects obtained by applying the terminal provided by the embodiment of the present invention are the same as those in the first embodiment and will not be described in detail here.

[0134] It should be noted that the memory may include random access memory (RAM) and may also include non-volatile memory (non-volatile memory), such as at least one disk storage. Similarly, the processor may also be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0135] Although the embodiments disclosed herein are as described above, the contents described herein are merely embodiments for facilitating understanding of the present invention and are not intended to limit the present invention. Any person skilled in the art may make any modifications and variations in the form and details of the embodiments without departing from the spirit and scope of the present invention. However, the scope of protection of the present invention shall remain subject to the scope defined by the appended claims.

Claims

1. A method for extracting image features, comprising: Acquire an object attribute recognition data set, wherein the object attribute recognition data set includes a plurality of object attribute recognition data, and the object attribute recognition data includes image data and text description data of target object attributes in the image data; Based on the trained image feature extraction model, feature extraction is performed on each object attribute recognition data in the object attribute recognition data set to obtain the region relationship attribute feature and the attribute relationship attribute feature of the target object in the image data of each object attribute recognition data; The image feature extraction model includes a visual area feature extraction unit, an attribute space information extraction unit, an attribute semantic information extraction unit and an attribute relationship extraction unit; The visual area feature extraction unit is used to perform preliminary feature extraction on the image data to obtain regional relationship attribute features and grid block local features; The attribute space information extraction unit is used to extract attribute space information from the local features of the grid block to obtain an attribute space embedding vector; The attribute semantic information extraction unit is used to extract attribute semantic information from the text description data to obtain an attribute semantic embedding vector; The attribute relationship extraction module is used to model attribute relationships based on the attribute space embedding vector and the attribute semantic embedding vector to obtain attribute relationship attribute features that model the spatial dimension attribute relationship and the semantic dimension attribute relationship.

2. The method according to claim 1, characterized in that The visual area feature extraction unit performs preliminary feature extraction on the image data to obtain regional relationship attribute features and grid block local features, including: Dividing the image data into a sequence of overlapping square grid blocks using a sliding window, and mapping the sequence of square grid blocks into an initial grid block embedding vector using a trainable linear mapping matrix; Adding a learnable global embedding vector before the first position of the initial grid block embedding vector to obtain a temporary grid block embedding vector; Adding a previous learnable grid block position embedding vector to the temporary grid block embedding vector to obtain a grid block embedding vector; Image region relationship modeling is performed on all element relationships in the grid block embedding vector using a preset modeling tool to obtain region relationship attribute features and grid block local features.

3. The method according to claim 1, characterized in that The attribute semantic information extraction unit extracts attribute semantic information from the text description data to obtain an attribute semantic embedding vector, which includes: Extracting word embedding vectors of different attributes in the text description data through a pre-trained word vector model to obtain attribute word embedding vectors; The attribute word embedding vector is mapped to the image domain using a learnable linear mapping matrix to obtain an attribute semantic embedding vector.

4. The method according to claim 1, wherein The attribute relationship extraction module models the attribute relationship based on the attribute space embedding vector and the attribute semantic embedding vector to obtain attribute relationship attribute features that model the spatial dimension attribute relationship and the semantic dimension attribute relationship, including: fusing the attribute space embedding vector and the attribute semantic embedding vector to obtain a fusion result, and adding a learnable attribute position embedding vector to the fusion result to obtain an attribute space semantic embedding vector; Attribute relationship modeling in spatial dimension and semantic dimension is performed on all element relationships in the attribute space semantic embedding vector by a preset modeling tool to obtain attribute relationship attribute features.

5. The method according to claim 1, wherein The process of training the image feature extraction model includes: Acquire an object attribute recognition training data set, the object attribute recognition training data set including a plurality of object attribute recognition training data, the object attribute recognition training data including image data, text description data of target object attributes in the image data, and label true values ​​of each attribute of the target object in the image data; Dividing all object attribute recognition training data in the object attribute recognition training data set into multiple groups to obtain multiple groups of object attribute recognition training data groups; A temporary image feature extraction model is trained based on the Mth group of object attribute recognition training data to obtain a parameter-updated image feature extraction model, and it is determined whether M is equal to a preset threshold. If so, it is determined whether the current parameter-updated image feature extraction model meets the loss value and reaches a convergence state. If so, the current parameter-updated image feature extraction model is used as the trained image feature extraction model. If M is not equal to the preset threshold, M is increased by 1, and the current parameter-updated image feature extraction model is used as a temporary image feature extraction model, and the temporary image feature extraction model is re-trained based on the Mth group of object attribute recognition training data. If the current parameter-updated image feature extraction model does not meet the loss value and reaches a convergence state, M is initialized to 1, and the current parameter-updated image feature extraction model is used as a temporary image feature extraction model, and the temporary image feature extraction model is re-trained based on the Mth group of object attribute recognition training data. Among them, the preset threshold is the total number of object attribute recognition training data groups in the object attribute recognition training data set, and the initial value of M is 1; when training is performed for the first time based on the object attribute recognition training data group, the temporary image feature extraction model is the initially created image feature extraction model.

6. The method according to claim 5, characterized in that During the training of the temporary image feature extraction model based on the object attribute recognition training data set, the process of optimizing the parameters of the temporary image feature model includes: inputting the region relationship attribute feature and the attribute relationship attribute feature of each object attribute recognition training data in the object attribute recognition training data group into the attribute classifier respectively to obtain a first attribute prediction score and a second attribute prediction score of each object attribute recognition training data; Calculating a first binary cross entropy loss value corresponding to the first attribute prediction score of each object attribute recognition training data based on the first attribute prediction score of each object attribute recognition training data, and calculating a second binary cross entropy loss value corresponding to the second attribute prediction score of each object attribute recognition training data based on the second attribute prediction score of each object attribute recognition training data; Obtaining a loss value of a feature extraction model loss function based on a first binary cross entropy loss value of a first attribute prediction score and a second binary cross entropy loss value of a second attribute prediction score of all the object attribute recognition training data; The temporary image feature model parameters are optimized based on the loss value of the feature extraction model loss function.

7. The method according to claim 6, characterized in that The expression of the binary cross entropy loss value is: Among them, L i The i-th binary cross entropy loss value represents the prediction score of the i-th attribute of the object attribute recognition training data, i∈(1,2), y i c represents the i-th predicted score value of the attribute classifier on the c-th attribute, Represents the true value of the label of the cth attribute of the input image data, ρ c =exp(-r c ) represents the weight of the cth attribute, r c represents the proportion of the cth attribute in all samples in the object attribute recognition training dataset, and σ(·) represents the sigmoid activation function; The feature extraction model loss function is: L=L1+λL2 Among them, λ represents the weight parameter.

8. An image feature extraction device, characterized in that: Including data acquisition module and feature extraction module; The data acquisition module is used to acquire an object attribute recognition data set, wherein the object attribute recognition data set includes a plurality of object attribute recognition data, and the object attribute recognition data includes image data and text description data of target object attributes in the image data; The feature extraction module is used to extract features from each object attribute recognition data in the object attribute recognition data set based on the trained image feature extraction model to obtain the region relationship attribute features and attribute relationship attribute features of the target object in the image data of each object attribute recognition data; The image feature extraction model includes a visual area feature extraction unit, an attribute space information extraction unit, an attribute semantic information extraction unit and an attribute relationship extraction unit; The visual area feature extraction unit is used to perform preliminary feature extraction on the image data to obtain regional relationship attribute features and grid block local features; The attribute space information extraction unit is used to extract attribute space information from the local features of the grid block to obtain an attribute space embedding vector; The attribute semantic information extraction unit is used to extract attribute semantic information from the text description data to obtain an attribute semantic embedding vector; The attribute relationship extraction module is used to model attribute relationships based on the attribute space embedding vector and the attribute semantic embedding vector to obtain attribute relationship attribute features that model the spatial dimension attribute relationship and the semantic dimension attribute relationship.

9. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image feature extraction method according to any one of claims 1 to 7 is implemented.

10. A terminal, characterized in that: include: a processor and a memory, wherein the memory is communicatively connected to the processor; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal performs the image feature extraction method according to any one of claims 1 to 7.

Images