Image semantic mapping method and device based on attribute interpretation

Abstract

The invention relates to the technical field of computer vision, and discloses an image semantic mapping method and device based on attribute interpretation, and the method comprises the steps: extracting an inherent feature, a first attribute feature and an image feature of a target image, converting the first attribute feature into an attribute vector, and converting the image feature into an image interpretation vector; determining a plurality of attribute interpretation vectors matched with the first attribute feature, and determining a semantic weight of each attribute interpretation vector; weighting the plurality of attribute interpretation vectors according to the semantic weight to obtain a weighted interpretation vector, and calculating a fusion attribute vector of the target image based on the weighted interpretation vector and the attribute vector; and determining a second attribute feature according to the fusion attribute vector, and taking the inherent feature of the target image and the second attribute feature as a semantic output result of the target image. According to the technical scheme provided by the invention, the accuracy of semantic interpretation of the target image can be improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular, to an image semantic mapping method and device based on attribute interpretation. Background Art

[0002] Currently, in the field of computer vision, compositional zero-shot learning enables a computer to quickly identify the semantic features of a target image through the combination of known attributes and objects.

[0003] In the practical application of compositional zero-shot learning, the definition and classification of attributes are relatively general, mostly single visual features. The same visual feature shows significant differences on different objects, but the recognition of attributes is relatively single, and there is often the problem of polysemy for the same attribute word, resulting in difficulty in aligning visual features and semantic information, and the semantic interpretation of attribute words is relatively broad, leading to a low accuracy of the semantic interpretation of images.

[0004] In view of this, how to improve the accuracy of the semantic interpretation of target images has become the focus of current research in the field of computer vision. Summary of the Invention

[0005] This application provides an image semantic mapping method and device based on attribute interpretation, which can improve the accuracy of the semantic interpretation of target images.

[0006] In the first aspect of this application, an image semantic mapping method based on attribute interpretation is provided. The method includes: obtaining a target image to be processed, extracting the inherent features, first attribute features, and image features of the target image, converting the first attribute features into attribute vectors, and converting the image features into image interpretation vectors; determining a plurality of attribute interpretation vectors that match the first attribute features, calculating the similarity between any one of the attribute interpretation vectors and the image interpretation vector, weighting each of the attribute interpretation vectors based on the similarity to obtain a weighted interpretation vector, and calculating a fusion attribute vector of the target image based on the weighted interpretation vector and the attribute vector; determining second attribute features according to the fusion attribute vector, and using the inherent features and second attribute features of the target image as the semantic output result of the target image.

[0007] In a possible implementation manner, extracting the inherent features, first attribute features, and image features of the target image, and converting the first attribute features into attribute vectors and the image features into image interpretation vectors includes: inputting the target image into a feature extraction network to generate basic features of the target image; embedding the basic features into the target image, and mapping the target image with the basic features into a first feature space, a second feature space, and a third feature space of the feature extraction network; generating the inherent features, first attribute features, and image features of the target image based on the mapping results, converting the first attribute features into attribute vectors, and converting the image features into image interpretation vectors.

[0008] In a possible implementation manner, determining a plurality of attribute interpretation vectors that match the first attribute features includes: based on a constructed attribute knowledge base, determining attribute texts that match the first attribute features, and obtaining an attribute phrase set of the attribute texts, the attribute phrase set including a plurality of attribute phrases, and the attribute phrases including a plurality of attribute words; traversing the attribute phrase set, and for any one of the attribute phrases in the attribute phrase set, converting each attribute word of the attribute phrase into an attribute word vector; taking the average value of each attribute word vector, and determining an attribute interpretation vector of the current attribute phrase based on the result of taking the average value to obtain a preset first number of attribute interpretation vectors.

[0009] In a possible implementation manner, constructing an attribute knowledge base in the following manner: obtaining a plurality of attribute texts, pre-classifying the plurality of attribute texts, and determining the attribute categories of each of the attribute texts according to the classification results; generating an attribute phrase set of the attribute texts based on each of the attribute texts and the name of the corresponding attribute category, the attribute phrase set including a preset first number of attribute phrases, and the attribute phrases including a preset second number of attribute words; constructing the attribute knowledge base according to the plurality of attribute texts and the attribute phrase sets corresponding to each of the attribute texts.

[0010] In a possible implementation manner, the attribute interpretation vector is represented in the following manner: Wherein, is the attribute interpretation vector, M is a preset second number of values, is the j-th attribute word vector from the i-th attribute phrase.

[0011] In a possible implementation, calculating the attribute similarity between any one of the attribute interpretation vectors and the image interpretation vector, and weighting each of the attribute interpretation vectors based on the attribute similarity includes: calculating the similarity between the image interpretation vector and each of the attribute interpretation vectors, and determining the semantic weight of each attribute interpretation vector according to the similarity; for any one of the attribute interpretation vectors, weighting the attribute interpretation vector based on the semantic weight corresponding to the attribute interpretation vector.

[0012] In a possible implementation, the attribute similarity and the semantic weight are represented as follows: where s ij is the attribute similarity, α ij is the semantic weight, v i is the image interpretation vector of the attribute phrase, and e ij is the attribute interpretation vector of the attribute phrase.

[0013] In a possible implementation, the weighted interpretation vector is represented as follows: where e′ i is the weighted interpretation vector, N is the number of attribute interpretation vectors, α ij is the semantic weight, and e ij is the attribute interpretation vector.

[0014] In a possible implementation, the fused attribute vector is represented as follows: x a = λ p w i +(1 - λ p )e′ i , where x a is the fused attribute vector, w i is the attribute vector, e′ i is the weighted interpretation vector, and λ p is the balance variable parameter.

[0015] In a possible implementation, determining the second attribute feature according to the fused attribute vector includes: mapping the fused attribute vector to a feature matching network, where the feature matching network includes multiple different attribute features; the feature matching network determines the attribute feature pointed to by the fused attribute vector as the second attribute feature of the target image.

[0016] In a possible implementation manner, the feature matching network is a trained feature network, where the feature matching network is trained as follows: Obtain a training set including a plurality of input images, map each input image to an image encoder to obtain an image feature vector, and determine the label words of each input image, and convert the label words into text feature vectors through word embedding; Divide the text feature vectors into positive sample vectors and negative sample vectors, and map the text feature vectors to the same vector space corresponding to the image feature vectors; Match and align the image feature vectors with the text feature vectors to shorten the distance between the positive sample vectors and the image feature vectors, and widen the distance between the negative sample vectors and the image feature vectors, so as to form a trained feature network.

[0017] The second aspect of this application provides an image semantic mapping device based on attribute interpretation, and the device includes: a feature extraction unit, configured to obtain a target image to be processed, extract the first attribute feature, intrinsic feature and image feature of the target image, and convert the first attribute feature into an attribute vector, and convert the image feature into an image interpretation vector; a feature processing unit, configured to determine a plurality of attribute interpretation vectors matching the first attribute feature, calculate the attribute similarity between any one of the first attribute interpretation vectors and the image interpretation vector, weight each of the attribute interpretation vectors based on the attribute similarity to obtain a weighted interpretation vector, and calculate the fused attribute vector of the target image based on the weighted interpretation vector and the attribute vector; a feature matching network, configured to determine the second attribute feature of the target image according to the fused feature vector, and use the intrinsic feature and the second attribute feature of the target image as the semantic output result of the target image.

[0018] The technical solution provided by one or more embodiments of this application refines and matches the attribute features of the target image, and adaptively adjusts the expression of the attribute features to improve the accuracy of semantic interpretation. Specifically, extract the attribute features and image features of the target image, weight the attribute interpretation vectors matching the attribute features according to the similarity with the image features, and fuse and express the obtained weighted interpretation vectors with the attribute features to achieve the accurate expression of the image adaptive attribute features, improve the consistency between the visual features and attribute features of the target image, and thus accurately express the semantic information of the target image.

[0019] It can be seen that the technical solution provided by this application can improve the accuracy of semantic interpretation of the target image. Description of the Drawings

[0020] To more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0021] Figure 1 Schematic diagram of the steps of an image semantic mapping method based on attribute interpretation provided for an embodiment of the present application;

[0022] Figure 2 Schematic diagram of the steps of initial feature extraction for a target image provided for an embodiment of the present application;

[0023] Figure 3 Schematic diagram of the steps of determining multiple matching attribute interpretation vectors provided for an embodiment of the present application;

[0024] Figure 4 Schematic diagram of the steps of adaptively weighting an attribute interpretation vector provided for an embodiment of the present application;

[0025] Figure 5 Schematic diagram of the steps of training a feature matching network provided for an embodiment of the present application;

[0026] Figures 6(a) and (b) are comparative schematic diagrams of the classification of multi-category attribute features provided for an embodiment of the present application;

[0027] Figure 7 Schematic diagram of a method for obtaining a weighted attribute vector with image adaptive attribute interpretation provided for an embodiment of the present application;

[0028] Figure 8 Schematic diagram of the structure of an image semantic mapping device based on attribute interpretation provided for an embodiment of the present application;

[0029] Figure 9 Schematic diagram of the structure of a computer device provided for an embodiment of the present application. Specific embodiments

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.

[0031] In addition, the descriptions involving "first", "second", etc. in this application are for descriptive purposes only and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more. In addition, the use of "based on" or "according to" means openness and inclusiveness, because the process, steps, calculations or other actions "based on" or "according to" one or more of the conditions or values ​​can be based on additional conditions or beyond the described values ​​in practice.

[0032] With the continuous development of the computer vision industry, combined zero-shot learning has been widely used. Its goal is to enable computers to quickly identify new categories that have not been seen by combining known attributes and objects, and to express the attribute characteristics and inherent characteristics of the object as semantic expressions. Among them, the inherent characteristics are the characteristics that are inherent to the object or scene itself and do not depend on external conditions. Such characteristics are usually applicable to most instances of the same type of objects. Attribute features are effective expressions of the additional properties of objects in the scene, reflecting the material, color, shape, state, function and other factors of the object, usually expressed in the form of adjectives. For example, the semantic expressions such as "ancient building" and "dilapidated house" have the inherent characteristics of "building" and "house" respectively, and the attribute characteristics of "ancient" and "dilapidated" respectively.

[0033] However, for the attribute expression of objects, there is usually a lack of deep semantic mining and classification. In related technologies, the definition and classification of attributes are relatively general, and most of them regard attributes as a single visual feature, while some attribute words may correspond to multiple visual expressions in different contexts. For example, "dirty water" and "dirty ground". Although the attribute features are both "dirty", the visual forms presented by the objects are different, which can easily lead to differences between visual expression and semantic understanding, and the accuracy of the semantic interpretation of the image is low. In addition, for the attribute expression of the same inherent object, its attribute characteristics may change with time and changes in conditions. For example, the semantic expression of "dry ground" may vary in degree of "dryness" depending on the scene, such as "dry ground" and "cracked ground". This simplified processing of attributes makes it difficult to effectively learn and generalize when facing complex attributes, reducing the accuracy of the semantic interpretation of the image.

[0034] In addition, in the task of combined zero-shot learning in related technologies, the model needs to make accurate predictions on unseen combinations of attributes and objects. However, when dealing with such cross-domain combinations, the language description is relatively concise, while the visual features are relatively complex and it is easy to ignore the characteristics attached to the objects, resulting in the inconsistency between visual performance and semantic understanding, and the inability to accurately match semantics and visual features, thereby leading to a low precision in the semantic interpretation of images. In view of this, one or more embodiments of the present application provide an image semantic mapping method and device based on attribute interpretation, which can solve the above problems and improve the precision of semantic interpretation of target images.

[0035] Please refer to Figure 1 , one embodiment of the present application provides an image semantic mapping method based on attribute interpretation, and this method may include the following multiple steps:

[0036] S1: Obtain a target image to be processed, extract the inherent features, first attribute features, and image features of the target image, convert the first attribute features into attribute vectors, and convert the image features into image interpretation vectors.

[0037] The above-mentioned inherent features and first attribute features can be understood as different types of semantic information in an object or a scene. Among them, the above-mentioned inherent features are the essential features of the object itself that are inherent and do not depend on external conditions, and are used to express the object itself. The above-mentioned first attribute features are a general expression of the additional properties of the object under a certain scene, and are used to initially express various elements such as the material, color, shape, state, and function of the object. Exemplarily, for target images with different degrees such as "dry ground" and "cracked ground", their inherent features are both "ground", and the first attribute feature is "dry".

[0038] The above-mentioned image features can be understood as image features jointly composed of the overall image features, object shape features, and object primitives of the input target image, and are used for a global abstract representation of the input image. The above-mentioned image features can reflect image information such as the contour and structural composition of the object, so as to more accurately determine the semantic information of the input image.

[0039] In this embodiment, the inherent features, first attribute features, and image features of the target image are extracted as the initial semantic information of the target image, and the above-mentioned respective features are converted into vector forms. Specifically, an image encoder or a pre-trained convolutional neural network can be selected to extract basic features from the target image, and the basic features are further mapped to different feature spaces to obtain inherent features, first attribute features, and image features with different semantics, and the first attribute features and image features are converted into fixed-dimensional vector forms through the linear layer or other encoders in the feature space to form an attribute vector and an image interpretation vector. Optionally, the basic features can also be further processed through specific network layers, such as fully connected layers or feature mapping layers, to obtain different feature information of the objects in the target image.

[0040] In this embodiment, through the preliminary detection of the input image, the inherent features, first attribute features, and image features of the target image are identified as the initial semantic information, and the first attribute features and image features are converted into an attribute vector and an image interpretation vector, which facilitates subsequent operations such as matching, weighting, or splicing of the attribute features in vector form to refine the expression of the attribute features and achieve multi-modal semantic alignment and understanding.

[0041] S3: Determine a plurality of attribute interpretation vectors that match the first attribute feature, calculate the attribute similarity between any one of the attribute interpretation vectors and the image interpretation vector, weight the respective attribute interpretation vectors based on the attribute similarity to obtain a weighted interpretation vector, and calculate the fusion attribute vector of the target image based on the weighted interpretation vector and the attribute vector.

[0042] Since the attribute vector of the first attribute feature detected according to the image is too general, it is necessary to determine a plurality of attribute interpretation vectors that match the first attribute feature. The above-mentioned attribute interpretation vectors can be understood as each refined feature vector of the current attribute vector, and each first attribute feature corresponds to a plurality of attribute interpretation vectors. Exemplarily, the above-mentioned attribute vector can represent the first attribute feature of "dry", and the corresponding attribute interpretation vectors can include feature vectors representing attributes such as "dry", "dried up", and "cracked". By refining the first attribute feature to form a plurality of attribute interpretation vectors, multiple possibilities are provided for the attribute semantic information of the target image, thereby facilitating the determination of more accurate semantic information of the target image.

[0043] In this embodiment, each attribute interpretation vector can be weighted according to the attribute similarity to form a weighted interpretation vector, so as to adaptively adjust the selection preference of the attribute interpretation vector. Specifically, calculate the similarity between each attribute interpretation vector and the image interpretation vector, and weight the attribute interpretation vectors according to the similarity of each attribute interpretation vector. Optionally, the similarity can be used as the weight value for weighting to weight the attribute interpretation vectors to form a weighted interpretation vector. By weighting the attribute interpretation vectors according to the similarity, the weight distribution can be adaptively adjusted according to the target image, improving the matching degree between the weighted interpretation vector and the target image, so as to obtain more accurate attribute features of the target image.

[0044] In this embodiment, according to the above-mentioned weighted interpretation vector and the attribute vector, the fused attribute vector of the target image is calculated. In order to make the weighted interpretation vector better fit the target image, it is fused with the initially detected attribute vector. Optionally, the weighted interpretation vector can be fused according to a preset fusion condition, or the fused attribute vector can be obtained through operations such as weighted summation or vector concatenation, so as to determine more accurate attribute features and improve the accuracy of semantic detection of the target image.

[0045] S5: Determine the second attribute feature according to the fused attribute vector, and use the inherent feature and the second attribute feature of the target image as the semantic output result of the target image.

[0046] The above-mentioned second attribute feature is the attribute feature represented by the fused attribute vector, and the attribute information of the above-mentioned second attribute feature is more accurate than that of the first attribute feature. Exemplarily, the input image is an image of a withered flower. The above-mentioned first attribute feature output is "dry", the above-mentioned second attribute feature output is "withered", and the above-mentioned inherent feature output is "flower". In view of this, it is more accurate to use the above-mentioned inherent feature and the second attribute feature as the semantic output result of the target image, and the semantic expression of the attribute feature is deeper and more accurate.

[0047] In this embodiment, the second attribute feature is determined according to the fused attribute feature obtained in step S3 and output in text form. Optionally, the fused attribute vector can be mapped to a specific second attribute feature through a decoder or a classifier, and the second attribute feature can be decoded into a natural language description. Optionally, the text output of the second attribute feature can also be generated by searching for the nearest neighbor text description, such as the nearest neighbor search in the word embedding space.

[0048] The technical solution provided by this embodiment extracts the attribute features and image features of the target image, weights the attribute interpretation vectors matching the attribute features according to the similarity with the image features, and fuses and expresses the obtained weighted interpretation vectors with the attribute features, so as to achieve the accurate expression of the image adaptive attribute features, improve the consistency of the visual features and attribute features of the target image, and thus accurately express the semantic information of the target image to improve the accuracy of the semantic interpretation of the target image.

[0049] Please refer to Figure 2 , in a possible implementation manner, based on the above step S1, extracting the inherent features, first attribute features and image features of the target image, and converting the first attribute features into attribute vectors and the image features into image interpretation vectors includes:

[0050] S101: Input the target image into the feature extraction network to generate the basic features of the target image.

[0051] S103: Embed the basic features into the target image, and map the target image with the basic features to the first feature space, second feature space and third feature space of the feature extraction network.

[0052] S105: Generate the inherent features, first attribute features and image features of the target image based on the mapping results, convert the first attribute features into attribute vectors, and convert the image features into image interpretation vectors.

[0053] In this embodiment, the basic features of the target image are generated through the feature extraction network to form the inherent features, first attribute features and image features of the target image. Exemplarily, a trained convolutional neural network is used as the feature extraction network to extract the basic features of the target image, and the target image with the extracted basic features is mapped to different feature spaces, where the basic features can be embedded into the target image through feature fusion or attention mechanism.

[0054] The above-mentioned first feature space, second feature space, and third feature space respectively represent different feature categories or descriptions. Among them, the above-mentioned first feature space reflects the inherent features of the object itself that do not depend on external conditions. The above-mentioned second feature space reflects the first attribute features with stronger semantics that focus on the environmental information or state information of the object. The above-mentioned third feature space reflects the image features jointly composed of the overall image features, object shape features, and object primitive features of the target image. The target image with the extracted basic features is mapped to different feature spaces, and each feature space can be implemented through different linear layers or encoders. For example, three linear layers of 512×300 are set as three mapping encoders to map the image to feature regions with different meanings, forming the above-mentioned inherent features, first attribute features, and image features. Further, the above-mentioned first attribute features are output as attribute vectors, and the image features are output as image interpretation vectors.

[0055] The technical solution provided by this embodiment of the present application extracts the initial semantic features of the inherent features, first attribute features, and image features of the target image. Since the features in vector form are more suitable for computing, matching, and generating tasks, the above-mentioned initial semantic features are represented in the form of vectors, which is convenient for further processing of the above-mentioned features.

[0056] Please refer to Figure 3 , in a possible implementation manner, determining multiple attribute interpretation vectors that match the first attribute features in step S3 includes:

[0057] S301: Based on the constructed attribute knowledge base, determine the attribute text that matches the first attribute feature, and obtain the set of attribute phrases of the attribute text. The set of attribute phrases includes multiple attribute phrases, and each attribute phrase includes multiple attribute words.

[0058] S303: Traverse the set of attribute phrases. For any attribute phrase in the set of attribute phrases, convert each attribute word of the attribute phrase into an attribute word vector.

[0059] S305: Take the mean of each attribute word vector, and determine the attribute interpretation vector of the current attribute phrase based on the result of taking the mean, so as to obtain multiple attribute interpretation vectors that match the first attribute feature.

[0060] In this embodiment, a plurality of attribute interpretation vectors matching the first attribute feature are determined through the constructed attribute knowledge base. Among them, the constructed attribute knowledge base includes a set of attribute phrases for deeper definition and classification of attributes. According to the attribute knowledge base, attribute texts matching the first attribute feature can be searched to obtain a set of attribute phrases including various attribute phrases under the current attribute text, so as to obtain a set of attribute phrases for various attribute interpretations related to the current first attribute feature.

[0061] Further, according to the set of matching attribute phrases, a plurality of attribute interpretation vectors matching the first attribute feature are determined. Among them, the above-mentioned attribute phrases include a plurality of attribute words. Each attribute word is converted into an attribute word vector, and then the average value of each attribute word vector is taken to obtain the attribute interpretation vector of the current attribute phrase, so as to obtain a plurality of attribute interpretation vectors matching the first attribute feature. For example, each attribute word can be converted into an attribute word vector by means of word embedding, so as to map the text information into a high-dimensional semantic space for vector calculation.

[0062] In this embodiment, an attribute knowledge base including detailed and comprehensive attribute phrases can be constructed in advance. Specifically, a plurality of attribute texts are obtained, and the plurality of attribute texts are pre-classified. According to the classification results, the attribute categories of each of the attribute texts are determined. Among them, the attribute categories can include physical form, visual performance, material, functional state, and weather environment. By pre-classifying the attribute texts, corresponding attribute phrases can be generated more standardly. Further, based on each of the attribute texts and the names of the corresponding attribute categories, a set of attribute phrases for the attribute text is generated. The set of attribute phrases includes a preset first number of attribute phrases, and each attribute phrase includes a preset second number of attribute words. Exemplarily, the attribute phrases can include attribute words such as attribute text, attribute description, and inherent feature text. For example, "dry means parched surface", each field is an attribute word, and each attribute phrase includes four attribute words. Further, according to the plurality of attribute texts and the sets of attribute phrases corresponding to each attribute text, an attribute knowledge base is constructed. The above-mentioned attribute knowledge base includes a plurality of sets of attribute phrases. By constructing the attribute knowledge base in advance, various attribute interpretations are provided for the first attribute feature, so as to re-select a more accurate attribute feature.

[0063] In this embodiment, the attribute interpretation vector can be represented in the following manner: Among them, is the attribute interpretation vector, M is the number of attribute word vectors, is the j-th attribute word vector in the i-th attribute phrase. It can be understood that the average value operation is performed on each attribute word vector of the current attribute phrase to obtain the attribute interpretation vector of the current attribute phrase.

[0064] Please refer to Figure 4 , in a possible implementation, based on the above step S3, calculating the attribute similarity between any one of the attribute interpretation vectors and the image interpretation vector, and weighting each of the attribute interpretation vectors based on the attribute similarity may include the following steps:

[0065] S311: Calculate the attribute similarity between the image interpretation vector and each of the attribute interpretation vectors, and determine the semantic weight of each attribute interpretation vector according to the attribute similarity.

[0066] S313: For any one of the attribute interpretation vectors, weight the attribute interpretation vector based on the semantic weight corresponding to the attribute interpretation vector.

[0067] In this implementation, weights are assigned to each attribute interpretation vector according to the attribute similarity between the image interpretation vector and each attribute interpretation vector. Optionally, the attribute similarity between each attribute interpretation vector can be calculated by cosine similarity or dot product.

[0068] Exemplarily, the attribute similarity is calculated in the manner of cosine similarity. According to s ij is the attribute similarity, v i is the image interpretation vector of the attribute phrase, and e ij is the attribute interpretation vector of the attribute phrase. Further, the semantic weight is determined according to the above attribute similarity as the weight value of the attribute interpretation vector. Optionally, the attribute similarity of each attribute interpretation vector can be used as the semantic weight of the current attribute interpretation vector.

[0069] Preferably, the softmax function is applied to the above attribute similarity to calculate the semantic weight. Specifically, α ij is the semantic weight. Since the exponential function exp(s ij ) is positive, the obtained semantic weight α ij is also positive.

[0070] The above semantic weight can be used to assign weights to each attribute interpretation vector, and sum the weighted attribute interpretation vectors to generate a weighted interpretation vector. Specifically, e′ i is the weighted interpretation vector, N is the number of attribute interpretation vectors, α ij is the semantic weight, and e ij is the attribute interpretation vector.

[0071] Further, the obtained weighted interpretation vector and the attribute vector obtained in S1 are fused to form a fused attribute vector. Specifically, the fused attribute vector is represented in the following manner: xa =λ p w i +(1-λ p )e′ i , where x a is the fusion attribute vector, w i is the attribute vector, e′ i is the weighted explanation vector, λ p is a balance variable parameter. Preferably, p Can be set to 0.7.

[0072] In one embodiment, based on the above step S5, determining the second attribute feature according to the fused attribute vector includes: mapping the fused attribute vector to a feature matching network, the feature matching network including multiple different attribute features; the feature matching network determines the attribute feature pointed to by the fused attribute vector as the second attribute feature of the target object.

[0073] In this embodiment, the feature matching network can be a deep learning-based classifier or a trained feature network. The feature matching network includes multiple different attribute features, which are used to match the input fused attribute vector with a preset attribute feature set to map the fused attribute vector to the most relevant attribute feature. The most relevant attribute feature is the attribute feature pointed to by the fused attribute vector, which is determined as the second attribute feature of the target image.

[0074] See also Figure 5 In one embodiment, optionally, the feature matching network is a trained feature network, wherein the feature matching network is trained in the following manner:

[0075] S10: obtaining a training set including a plurality of training images, mapping each training image to an image encoder to obtain an image feature vector, determining a label word for each training image, and converting the label word into a text feature vector through word embedding;

[0076] S11: dividing the text feature vector into a positive sample vector and a negative sample vector, and mapping the text feature vector to the same vector space as that corresponding to the image feature vector;

[0077] S12: Matching and aligning the image feature vector with the text feature vector to shorten the distance between the positive sample vector and the image feature vector, and to increase the distance between the negative sample vector and the image feature vector, so as to form a trained feature network.

[0078] In this embodiment, each image in the training set is input into an image encoder and mapped to a feature space through the image encoder to obtain the image feature vectors of each training image. At the same time, a label word is set for each training image, and the label word is converted into a text feature vector through word embedding encoding. The text feature vector is divided into positive and negative samples and mapped to the same vector space as the image feature vector corresponding to the current training image for loss calculation. During the training process, the image feature vector is used as the sample to be learned. By continuously reducing the distance between the sample to be learned and the positive sample and increasing the distance between the sample to be learned and the negative sample, the image feature vector can be made as close as possible to the positive sample vector and as far as possible from the negative sample vector, so as to achieve the effect of image-text matching, realize the consistency between visual performance and semantic understanding, accurately match semantics and visual features, and thus improve the accuracy of semantic interpretation of images.

[0079] Please refer to FIGS. 6(a) and 6(b). FIG. 6(a) shows the classification of multi-class attribute features in an untrained feature network, and FIG. 6(b) shows the classification of multi-class attribute features in a trained feature matching network. Taking the classification comparison of three similar attributes as an example, the categories of attribute features include "broken", "chipped", and "cracked". It can be seen from the figure that the vector distributions of attribute texts of the same category in the trained feature matching network are relatively concentrated, which can avoid interference from other category attributes and can point to the most similar attribute text when performing vector mapping on the feature matching network, further improving the accuracy of semantic interpretation of the target image.

[0080] This application provides an embodiment of applying the above-mentioned image semantic mapping method based on attribute interpretation, and this embodiment is carried out according to the following steps:

[0081] Step 1: Construct an attribute knowledge base for attribute features.

[0082] Specifically, construct an attribute vocabulary including multiple attribute texts, generate 20 attribute phrase groups for each attribute text. The above-mentioned attribute phrase groups include two attribute words and an attribute constraint such as 'ancient means' to form an attribute phrase group including four fields. Each attribute text and the corresponding 20 attribute phrase groups form an attribute knowledge base.

[0083] Optionally, a large language model can be used to collect and expand the attribute phrases. Specifically, the attribute text is pre-classified to determine which attribute category the attribute word belongs to. The above-mentioned attribute categories include physical form category, visual performance category, material category, functional state category, and weather environment category. The attribute word and its belonging attribute category are used as prompt words and input into the large language model, adding 'ancient means' as a constraint, and generating attribute phrases in the form of 'constraint word' 'adjective' 'noun', such as "Dry means parched surface".

[0084] Exemplarily, taking the attribute text "dry" as an example, its attribute phrases can include: "Dry means parched surface", "Dry means cracked texture", "Dry means brittle structure", "Dry means dehydrated appearance", "Dry means flaky layers", "Dry means desiccated material", "Dry means no moisture state", "Dry means arid touch", "Dry means powdery residue", "Dry means withered look", "Dry means dusty covering", "Dry means arid complexion", "Dry means rough feel", "Dry means friable consistency", "Dry means husky sound", "Dry means shriveled form", "Dry means hardened facade", "Dry means thirsty aspect", "Dry means baked quality", "Dry means lifeless nature".

[0085] Step 2: For the target image to be detected, obtain a weighted attribute vector with image-adaptive attribute interpretation.

[0086] Specifically, please refer to Figure 7, identify the inherent features, first attribute features, and image interpretation vectors of the target image, search the above-mentioned attribute knowledge base to obtain the set of attribute phrases corresponding to the first attribute features, and convert each attribute phrase in the set of attribute phrases into an attribute interpretation vector. Calculate the cosine similarity between each attribute interpretation vector and the image interpretation vector, and determine the semantic weight of the current attribute interpretation vector according to the cosine similarity to obtain a weighted attribute vector with image-adaptive attribute interpretations.

[0087] Exemplarily, taking an image of a withered flower as the target image, its first attribute feature is "dry". Search the attribute knowledge base constructed in step one for the set of attribute phrases that match "dry". For example, the set of attribute phrases for the above "dry" includes 20 attribute phrases. Further, represent each attribute phrase as a vector, convert each attribute phrase into an attribute interpretation vector, and use the cosine similarity calculation method to calculate the attribute similarity between any attribute interpretation vector and the image interpretation vector according to the image interpretation vector, and apply the softmax function to the above attribute similarity to calculate the semantic weight, assign weights to each attribute interpretation vector, and sum the weighted attribute interpretation vectors to generate a weighted interpretation vector.

[0088] Step 3: Calculate the fused attribute vector of the target image and map the fused attribute vector to the feature matching network to obtain the final attribute features.

[0089] Specifically, based on the weighted interpretation vector and the attribute vector corresponding to the first attribute feature, calculate the fused attribute vector of the target image, and obtain the attribute features corresponding to the fused attribute vector according to the feature matching network. Exemplarily, taking "withered flower" as an example, the first attribute feature is "dry", and the attribute feature output after fusion is "withered", and it is used as the output result together with the inherent feature, such as "withered flower", which means "a withered flower".

[0090] Through the semantic detection of the target image in steps one to three, compared with traditional image semantic detection, the interpretation of attribute features is more accurate, and the accuracy of the semantic interpretation of the target image is higher.

[0091] In another embodiment of the present application, not only an attribute knowledge base is constructed for attribute features, but also an object knowledge base is constructed for inherent features. While ensuring the accuracy of attribute features, the accuracy of detecting inherent features is also improved. For example, for the inherent feature of "flower", it can be enriched to inherent features in the variety directions of "rose", "Chinese rose", etc. The further functional description of this embodiment is the same as that of the embodiment that only constructs an attribute knowledge base for attribute features, and will not be elaborated here.

[0092] Please refer to Figure 8, this application also provides an image semantic mapping device based on attribute interpretation, and the device includes:

[0093] A feature extraction unit 100, which acquires a target image to be processed, extracts the inherent features, first attribute features, and image features of the target image, converts the first attribute features into attribute vectors, and converts the image features into image interpretation vectors.

[0094] A feature processing unit 200, which is configured to determine a plurality of attribute interpretation vectors that match the first attribute features, calculate the attribute similarity between any one of the attribute interpretation vectors and the image interpretation vectors, weight each of the attribute interpretation vectors based on the attribute similarity to obtain a weighted interpretation vector, and calculate the fused attribute vector of the target image based on the weighted interpretation vector and the attribute vector.

[0095] A feature matching unit 300, which is configured to determine the second attribute features of the target image according to the fused feature vector, and use the inherent features and the second attribute features of the target image as the semantic output result of the target image.

[0096] Wherein,

[0097] In one embodiment, the feature extraction unit 100 is specifically configured to perform preliminary detection on the target image, extract basic features from the target image, and further map the basic features to different feature spaces to obtain inherent features, first attribute features, and image features with different semantics, and convert each of the above features into a vector form. This facilitates subsequent matching or weighting operations on the attribute features to refine the expression of the attribute features and achieve multi-modal semantic alignment and understanding.

[0098] In one embodiment, the feature processing unit 200 further includes an attribute knowledge base, and the attribute knowledge base includes a preset first number of attribute texts and the set of attribute phrases corresponding to each of the attribute texts, wherein: the attribute knowledge base is used to find the attribute texts that match the first attribute features, and obtain the set of attribute phrases corresponding to the attribute texts, and the set of attribute phrases includes a preset first number of attribute phrases.

[0099] In one embodiment, the feature processing unit 200 is specifically configured to obtain multiple attribute phrases corresponding to the first attribute feature, convert each attribute phrase into an attribute interpretation vector, determine the similarity of each attribute interpretation vector according to the first attribute interpretation vector and the image interpretation vector, determine the semantic weight of each attribute interpretation vector according to the similarity, weight the attribute interpretation vector based on the semantic weight corresponding to the attribute interpretation vector to obtain multiple weighted interpretation vectors, and calculate the fused attribute vector of the target image based on the weighted interpretation vector and the attribute vector;

[0100] In one embodiment, the feature matching unit 300 is specifically configured to map the fused attribute vector to a feature matching network, where the feature matching network includes multiple attribute features, determine the attribute feature pointed to by the fused attribute vector as the second attribute feature of the target object, where the second attribute feature is the final attribute feature, and use the inherent feature and the second attribute feature of the target image as the semantic output result of the target image.

[0101] The further function descriptions of the above-mentioned modules or units are the same as those in the corresponding embodiments above, and will not be repeated here.

[0102] The image semantic mapping device based on attribute interpretation in the embodiments of the present application is presented in the form of functional units. Here, the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory that execute one or more software or fixed programs, or other devices that can provide the above functions.

[0103] Please refer to Figure 9 , Figure 9 which is a schematic structural diagram of a computer device provided in the embodiments of the present application. As Figure 9 shown, the computer device includes: one or more processors 10, a memory 20, and an interface for connecting each component, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some alternative embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (such as an array of servers, a set of blade servers, or a multi-processor system). Figure 9Taking a processor 10 as an example.

[0104] The processor 10 can be a central processing unit, a network processor, or a combination thereof. Among them, the processor 10 can further include a hardware chip. The above-mentioned hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above-mentioned programmable logic device can be a complex programmable logic device, a field-programmable gate array, a generic array logic, or any combination thereof.

[0105] Among them, the memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiments.

[0106] The memory 20 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the computer device, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 can optionally include a memory remotely set relative to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0107] The memory 20 can include a volatile memory, such as a random access memory; the memory can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid-state drive; the memory 20 can also include a combination of the above types of memories.

[0108] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or communication networks.

[0109] The embodiments of the present application also provide a computer-readable storage medium. The methods according to the embodiments of the present application can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented as computer code that is originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the methods described herein can be stored as such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the methods shown in the above embodiments are implemented.

[0110] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations all fall within the scope defined by the appended claims.

[0111] The devices, modules, or units illustrated in the above embodiments can be specifically implemented by a computer chip or an entity, or by a product with a certain function. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0112] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0113] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses, and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or one or more of the blocks.

[0115] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or one or more of the blocks.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or one or more of the blocks.

[0117] It should also be noted that the term "including", "comprising", or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity, or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, commodity, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, commodity, or device including the said element.

[0118] Each embodiment in this specification is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the apparatus embodiments, since they are basically similar to the method embodiments, they are described relatively simply, and the relevant parts can be referred to the description of the method embodiments.

[0119] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

[0120] Although the embodiments of the present application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations fall within the scope defined by the appended claims.

Claims

1. An image semantic mapping method based on attribute interpretation, characterized in that: The method comprises: Acquire a target image to be processed, extract the inherent features, first attribute features and image features of the target image, convert the first attribute features into an attribute vector, and convert the image features into an image interpretation vector; Determine a plurality of attribute explanation vectors matching the first attribute feature, and calculate the attribute similarity between any of the attribute explanation vectors and the image explanation vector, weight the attribute explanation vectors based on the attribute similarity to obtain a weighted explanation vector, and calculate a fused attribute vector of the target image based on the weighted explanation vector and the attribute vector; A second attribute feature is determined according to the fused attribute vector, and the inherent feature of the target image and the second attribute feature are used as the semantic output result of the target image.

2. The method according to claim 1, characterized in that Extracting the inherent features, the first attribute features and the image features of the target image, converting the first attribute features into an attribute vector, and converting the image features into an image interpretation vector comprises: Inputting the target image into a feature extraction network to generate basic features of the target image; Embedding the basic features into the target image, and mapping the target image with the basic features to the first feature space, the second feature space and the third feature space of the feature extraction network; Based on the mapping result, the intrinsic features, first attribute features and image features of the target image are generated, the first attribute features are converted into attribute vectors, and the image features are converted into image interpretation vectors.

3. The method according to claim 1, characterized in that Determining a plurality of attribute explanation vectors matching the first attribute feature comprises: Based on the constructed attribute knowledge base, determine the attribute text matching the first attribute feature, and obtain the attribute phrase set of the attribute text, the attribute phrase set includes multiple attribute phrases, and the attribute phrase includes multiple attribute words; Traversing the attribute phrase set, for any attribute phrase in the attribute phrase set, converting each attribute word of the attribute phrase into an attribute word vector; The attribute word vectors are averaged, and the attribute explanation vector of the current attribute word group is determined based on the average result to obtain a plurality of attribute explanation vectors matching the first attribute feature.

4. The method according to claim 3, characterized in that The attribute knowledge base is constructed as follows: Acquire multiple attribute texts, pre-classify the multiple attribute texts, and determine the attribute category of each of the attribute texts according to the classification result; Based on each of the attribute texts and the name of the corresponding attribute category, generating an attribute phrase set of the attribute text, the attribute phrase set including a preset first number of attribute phrases, and the attribute phrases including a preset second number of attribute words; The attribute knowledge base is constructed according to the plurality of attribute texts and the attribute phrase sets corresponding to the respective attribute texts.

5. The method according to claim 3, characterized in that: The attribute interpretation vector is represented as follows: in, is the attribute explanation vector, M is the number of attribute word vectors, is the jth attribute word vector from the i-th attribute word group.

6. The method according to claim 1, characterized in that Calculating the attribute similarity between any of the attribute interpretation vectors and the image interpretation vector, and weighting the attribute interpretation vectors based on the attribute similarity comprises: Calculating the attribute similarity between the image interpretation vector and each of the attribute interpretation vectors, and determining the semantic weight of each attribute interpretation vector according to the attribute similarity; For any of the attribute interpretation vectors, the attribute interpretation vector is weighted based on the semantic weight corresponding to the attribute interpretation vector.

7. The method according to claim 6, characterized in that The attribute similarity and the semantic weight are expressed in the following manner: Among them, s ij is the attribute similarity, α ij is the semantic weight, v i is the image interpretation vector of the attribute phrase, e ij The attribute explanation vector for the attribute phrase.

8. The method according to claim 1 or 6, characterized in that: The weighted interpretation vector is expressed as follows: Among them, e′ i is the weighted explanation vector, N is the number of attribute explanation vectors, α ij is the semantic weight, e ij Interpret the vector for the attribute.

9. The method according to claim 1, characterized in that: The fused attribute vector is represented as follows: x a =λ p w i +(1-l p )e′ i Among them, x a is the fusion attribute vector, w i is the attribute vector, e′ i is the weighted explanation vector, λ p is the balance variable parameter.

10. The method according to claim 1, characterized in that Determining the second attribute feature according to the fused attribute vector includes: Mapping the fused attribute vector to a feature matching network, wherein the feature matching network includes a plurality of different attribute features; The feature matching network determines the attribute feature pointed to by the fused attribute vector as the second attribute feature of the target image.

11. The method according to claim 10, characterized in that The feature matching network is a trained feature network, wherein the feature matching network is trained in the following manner: Obtain a training set including a plurality of training images, map each training image to an image encoder to obtain an image feature vector, determine a label word for each training image, and convert the label word into a text feature vector through word embedding; The text feature vector is divided into a positive sample vector and a negative sample vector, and the text feature vector is mapped to the same vector space as the image feature vector; The image feature vector is matched and aligned with the text feature vector to shorten the distance between the positive sample vector and the image feature vector and to increase the distance between the negative sample vector and the image feature vector, so as to form a trained feature network.

12. An image semantic mapping device based on attribute interpretation, characterized in that: The device comprises: A feature extraction unit, used for acquiring a target image to be processed, extracting an inherent feature, a first attribute feature and an image feature of the target image, converting the first attribute feature into an attribute vector, and converting the image feature into an image interpretation vector; a feature processing unit, configured to determine a plurality of attribute interpretation vectors matching the first attribute feature, calculate an attribute similarity between any of the attribute interpretation vectors and the image interpretation vector, weight the attribute interpretation vectors based on the attribute similarity to obtain a weighted interpretation vector, and calculate a fused attribute vector of the target image based on the weighted interpretation vector and the attribute vector; The feature matching unit is used to determine the second attribute feature of the target image according to the fused feature vector, and use the inherent feature and the second attribute feature of the target image as the semantic output result of the target image.

13. The image semantic mapping device based on attribute interpretation according to claim 12, characterized in that: The feature processing unit includes an attribute knowledge base, the attribute knowledge base includes a preset first number of attribute texts and a set of attribute phrases corresponding to each of the attribute texts, wherein: The attribute knowledge base is used to search for attribute text matching the first attribute feature and obtain an attribute phrase set corresponding to the attribute text, wherein the attribute phrase set includes a preset first number of attribute phrases.

Images