A combination zero-shot learning method based on a concept collaboration network and a device thereof

Abstract

The application discloses a kind of based on concept collaborative network's combined zero sample learning method and its device, comprising: obtaining original image to be measured;Using the feature extractor of trained concept collaborative network to process original image to be measured, obtain feature;Using the state encoder of trained concept collaborative network to process the feature of original image to be measured, obtain state feature;Using the object encoder of trained concept collaborative network to process the feature of original image to be measured, obtain object feature;Using the concept interaction module of trained concept collaborative network to process feature, state feature and object feature;The self-attention feature of feature, the self-attention feature of state feature and the self-attention feature of object feature of original image to be measured are spliced;Using the learning module of trained concept collaborative network to learn combined feature spliced, obtain the classification result of original image to be measured.The application can enhance recognition performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image classification, and particularly relates to a combined zero-shot learning method based on a concept collaboration network and a device thereof. BACKGROUND

[0002] The combined zero-shot learning aims to identify new state-object combinations that never appear in the training data, because the object represents a specific entity, and the state represents a brief description in semantics, so that the combined zero-shot learning is still a challenging task;

[0003] In the prior art, the visual appearance changes when the same state is combined with different objects, and the existing combined zero-shot learning focuses on regarding this problem as a supervised classification task, and learning two classifiers to respectively preset the state and the object, so as to directly mine and utilize the two concepts of state and object from the original visual features, and ignore the potential relationship between the state and the object, so that the classifier cannot completely separate the original features into two different concepts, and the final classification accuracy is limited.

[0004] Therefore, it is urgent to improve the above-mentioned defects existing in the prior art. SUMMARY

[0005] In order to solve the above-mentioned problems existing in the prior art, the application provides a combined zero-shot learning method based on a concept collaboration network and a device thereof. The technical problems to be solved by the application are realized by the following technical solutions:

[0006] In a first aspect, the application provides a combined zero-shot learning method based on a concept collaboration network, comprising:

[0007] Obtaining a to-be-tested original image of a state-object combination;

[0008] Using a feature extractor of a trained concept collaboration network to process the to-be-tested original image to obtain a feature of the to-be-tested original image;

[0009] Using a state encoder of the trained concept collaboration network to process the feature of the to-be-tested original image to obtain a state feature;

[0010] Using an object encoder of the trained concept collaboration network to process the feature of the to-be-tested original image to obtain an object feature;

[0011] Using a concept interaction module of the trained concept collaboration network to process the feature of the to-be-tested original image, the state feature and the object feature to respectively obtain a self-attention feature of the feature of the to-be-tested original image, a self-attention feature of the state feature and a self-attention feature of the object feature;

[0012] The self-attention feature of the object feature of the to-be-tested original image is spliced to obtain a combined feature;

[0013] The learning module of the trained concept collaboration network is used for learning the combined feature to obtain a classification result of the to-be-tested original image.

[0014] In a second aspect, the present application further provides a combined zero-shot learning device based on a concept collaboration network, comprising:

[0015] An image acquisition module is configured to acquire a to-be-tested original image of a state-object combination.

[0016] An image processing module one is configured to use a feature extractor of a trained concept collaboration network to process the to-be-tested original image to obtain a feature of the to-be-tested original image.

[0017] An image processing module two is configured to use a state encoder of a trained concept collaboration network to process the feature of the to-be-tested original image to obtain a state feature.

[0018] An image processing module three is configured to use an object encoder of a trained concept collaboration network to process the feature of the to-be-tested original image to obtain an object feature.

[0019] An image processing module four is configured to use a concept interaction module of a trained concept collaboration network to process the feature, the state feature and the object feature of the to-be-tested original image to obtain a self-attention feature of the feature of the to-be-tested original image, a self-attention feature of the state feature and a self-attention feature of the object feature.

[0020] An image processing module five is configured to splice the self-attention feature of the feature of the to-be-tested original image, the self-attention feature of the state feature and the self-attention feature of the object feature to obtain a combined feature.

[0021] A result acquisition module is configured to use a learning module of a trained concept collaboration network to learn the combined feature to obtain a learning result of the to-be-tested original image.

[0022] The present application has the following advantages:

[0023] The present application provides a combined zero-shot learning method and device based on a concept collaboration network, which on one hand proposes a new concept collaboration network to mine the concepts of states and objects, and uses the network to perfect original visual features to identify visible and invisible categories; on the other hand, proposes a concept interaction module to analyze the potential correlation between states and objects, and simultaneously rebalances the proportion of states and objects in the original visual features, thereby enhancing the distinguishability of the combined features.

[0024] The application will be described in further detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a flow chart of a combination zero-shot learning method based on a concept collaborative network provided by an embodiment of the application;

[0026] Figure 2 is a schematic diagram of a concept collaborative network architecture provided by an embodiment of the application;

[0027] Figure 3 is a schematic diagram of a comparison between the method of the application and the method of the prior art provided by an embodiment of the application;

[0028] Figure 4 is a schematic diagram of a classification result provided by an embodiment of the application;

[0029] Figure 5 is a schematic diagram of a combination zero-shot learning device based on a concept collaborative network provided by an embodiment of the application. DETAILED DESCRIPTION

[0030] The application will be described in further detail below with reference to the drawings and embodiments, but the embodiments of the application are not limited thereto.

[0031] In the prior art, the goal of combination zero-shot learning is to recognize new combinations of known concepts. Some methods recognize combinations by separately learning object and state classifiers, or directly learn joint state-object representations. It can be divided into classification-based methods, methods based on mining internal relations and generation-based methods.

[0032] Among them, the classification-based method decomposes the state and object to train the classifier respectively, and makes prediction. The article “Inferring analogous attributes” published by Chao-Yeh et al., and the article “From red wine to red tomato: Composition with context” published by Ishan Misra et al., and the article “Task-driven modular networks for zero-shot compositional learning” published by Senthil Purushwalkam et al. propose a task-driven modular architecture, which represents the recognition process as a combination of subtasks, and trains a set of small networks to solve the state and object respectively.

[0033] Existing approaches based on mining intrinsic connections are either considering that the meaning of a state has an intrinsic connection with the object it describes; Tushar Nagarajan et al. in the article “Attributes as operators: factorizing unseen attribute-object compositions” and Yong-Lu Li et al. in the article “Symmetry and group in attribute-object compositions”, which consider states as transformations in the object vector space; inspired by group theory, the transformation process is modeled using the principle of symmetry. Compared to the classifier-based approach, the above approaches apply explicit regularization to attributes, but ignore useful dependencies between concepts.

[0034] Existing approaches based on generation; Zhixiong Nan et al. in the article “Recognizing unseen attribute-object pair with generative model” and Kun Wei et al. in the article “Adversarial fine-grained composition learning for unseen attribute-object recognition”, propose a set of generative-based methods that generate samples from semantic information and convert the composition zero-shot learning into a traditional supervised classification problem, which can combine and enhance different attribute-object pair relationships in a common embedding space.

[0035] In summary, in the prior art, because the data available in the combined zero-shot learning setting is limited, and existing deep methods are largely data-driven to mine the correlation between instances, it is difficult to generalize the model to recognize unseen combinations in reality. In addition, the interaction between states and objects cannot be quantified, resulting in different contexts in different state-object combinations, which seriously hinders the successful recognition of new combinations. Existing combined zero-shot learning work focuses on treating this problem as a general supervised classification task and learning two classifiers to predict states and objects respectively, and their purpose is to directly mine and utilize the concepts of states and objects from the original visual features, ignoring the interaction between visual features. The classifier cannot completely separate the original features into two distinct concepts, thereby limiting the final classification accuracy. In addition, other methods aim to learn a joint state-object representation and embed it into a common embedding space, in which the composition and visual features can be projected to reduce the distance between them, such as Euclidean distance; However, the above methods only consider the combination as an entity, without considering the proportion of concepts in the original visual features, which cannot be generalized to recognize and distinguish similar combinations in the test, resulting in poor robustness and weak generalization ability of the model.

[0036] Therefore, the present application provides a combined zero-shot learning method based on a concept collaboration network, which can improve the original visual features and enhance the distinguishability of the combination representation by learning the joint compatibility of the encoded states and the encoded objects.

[0037] Please refer to Figures 1-4 shown, Figure 1 is a flowchart of the combined zero-shot learning method based on the concept collaboration network provided by the embodiments of the present application, Figure 2 is a schematic diagram of the concept collaboration network architecture provided by the embodiments of the present application, Figure 3 is a schematic diagram comparing the method of the present application with the prior art method, Figure 4 is a schematic diagram of the classification result, and the combined zero-shot learning method based on the concept collaboration network provided by the present application aims to recognize new combinations composed of multiple semantic concepts in images, i.e. new combinations of states and concepts, which comprises:

[0038] S101, obtaining a to-be-tested original image of a state-object combination.

[0039] S102, processing the to-be-tested original image using a feature extractor of a trained concept collaboration network to obtain the features of the to-be-tested original image.

[0040] S103, processing the features of the to-be-tested original image using the state encoder of the trained concept collaboration network to obtain state features.

[0041] S104, processing the features of the to-be-tested original image using the object encoder of the trained concept collaboration network to obtain object features.

[0042] Specifically, considering that the entanglement between the state and the object in the to-be-tested original image exacerbates the misclassification problem, the concept collaboration network is trained to separate the state and the object from the visual features of the to-be-tested original image, and to use the state concept and the object concept to improve the original visual features, so as to identify visible and invisible images.

[0043] The feature extractor of the trained concept collaboration network is used to extract the features of the to-be-tested original image, the state encoder is used to extract the state features, and the object encoder is used to extract the object features, so that the visual features of the state and the object in the to-be-tested original image are separated, the expression of the obtained state features is The expression of the obtained object features is Where x is the features of the to-be-tested original image, h s is the state features, h o is the object features, is the state encoder, is the object encoder.

[0044] S105, processing the features of the to-be-tested original image, the state features and the object features using the concept interaction module of the trained concept collaboration network to obtain the self-attention features of the features of the to-be-tested original image, the self-attention features of the state features and the self-attention features of the object features, respectively.

[0045] S106, splicing the self-attention features of the features of the to-be-tested original image, the self-attention features of the state features and the self-attention features of the object features to obtain combined features.

[0046] S107, learning the combined features using the learning module of the trained concept collaboration network to obtain the classification result of the to-be-tested original image.

[0047] Specifically, please continue to refer to Figures 1-3As shown, the present application provides a combination zero-shot learning method based on a concept collaborative network. First, a feature extractor of the trained concept collaborative network is used to process a to-be-tested original image to obtain a feature of the to-be-tested original image. Second, a state encoder and an object encoder of the trained concept collaborative network are used to process the feature of the to-be-tested original image to obtain a state feature and an object feature. This process can effectively mine state concepts and object concepts. Third, a concept interaction module of the trained concept collaborative network is used to process the feature of the to-be-tested original image, the state feature, and the object feature to obtain a self-attention feature of the feature of the to-be-tested original image, a self-attention feature of the state feature, and a self-attention feature of the object feature. This process, based on the concept interaction module, can pay more attention to potential links between state concepts and object concepts and rebalance the proportion between states and objects, further helping to improve the visual features of the original image and making the visual features of the original image more discriminative.

[0048] It should be noted that, Figure 2 The embodiments shown only schematically show the order of processing of each module in the trained concept collaborative network and do not represent the actual structure. Figure 3 The embodiments shown only schematically show a schematic diagram of the difference between the prior art and the learning method provided by the present application, as Figure 3 As shown in (a) of the prior art, the state feature and the object feature are directly obtained based on the original image feature, and the synthetic feature is directly obtained, as Figure 3 As shown in (b) of the prior art, after obtaining the state feature and the object feature, the learning method provided by the present application further processes them using the concept interaction module, rebalances the proportion between states and objects, and helps to improve the visual features of the original image, making the visual features of the original image more discriminative.

[0049] In an optional embodiment of the present application, the concept collaborative network is trained.

[0050] S201, obtain a state set S and an object set O, and generate a state-object combination pair C according to a state s in the state set S and an object o in the object set O, and the expression is:

[0051] C=SxO={(s,o)|s∈S,o∈O}.

[0052] Specifically, a plurality of possible combination pairs C are combined according to a state s in the state set S and an object o in the object set O.

[0053] S202, obtain a training set Γ according to the state-object combination pair, and the expression is:

[0054]

[0055] wherein x i is an image sample, χ is an image in the training set, c i ∈ C s is a subset of state-object combinations, i is the i-th image, and N is the number of images in the training set.

[0056] S203, input the training set into the concept collaboration network, update the loss function parameters of the concept collaboration network until the loss function value of the concept collaboration network reaches the minimum, and obtain the loss function parameters of the concept collaboration network; wherein the loss function of the concept collaboration network comprises a first loss function and a second loss function, the first loss function comprises a state encoder loss function and an object encoder loss function, and the second loss function comprises a loss function of a concept interaction module.

[0057] S204, constructing the trained concept collaboration network according to the loss function parameters of the state encoder and the object encoder.

[0058] In an optional embodiment of the present application, the expression of the first loss function l cls is as follows:

[0059] l cls = αl s + βl o ;

[0060] wherein α and β are influence factors for balancing the influence of the loss function, l S is the loss function of the state encoder, and l o is the loss function of the object encoder.

[0061] In an optional embodiment of the present application, the expression of the loss function l S of the state encoder is as follows:

[0062]

[0063] The expression of the loss function l o of the object encoder is as follows:

[0064]

[0065] wherein τ is a temperature parameter for balancing the loss generated by the scaling model, log is a logarithm, exp(·) is an exponential function, h s is a state feature, h o is an object feature, s' is a state in the state set, and o' is an object in the object set.

[0066] It should be noted that hs and h o The sensitive information of the classifier including state and object recognition respectively extracts discriminative concepts of the state and the object respectively by using distance metric learning as a constraint condition, and defines a state loss function and an object loss function.

[0067] In an optional embodiment of the present application, in order to enhance the distinguishability of the composed features, the present application aims to analyze the potential correlation between the state concept and the object concept, and balance the proportion of the state concept and the object concept in the original visual features, and uses the concept interaction module of the trained concept cooperation network to process the features, the state features and the object features of the to-be-tested original image, to obtain the self-attention features of the features, the self-attention features of the state features and the self-attention features of the object features of the to-be-tested original image respectively.

[0068] S301, projecting the features of the to-be-tested original image into a scaled dot product attention space to obtain the dot product attention of the features of the to-be-tested original image, and the expression is:

[0069]

[0070] wherein q x is a query vector, k x is a key vector, v x is a value vector, and are learnable matrices, and x is the features of the to-be-tested original image.

[0071] S302, projecting the state features into a scaled dot product attention space to obtain the dot product attention of the state features, and the expression is:

[0072]

[0073] wherein q s is a query of the state, k s is a key of the state, v s is a value of the state, ψ(·) is a state encoder, h s is the state features, and are transformation matrices from h s to q s , k s and v s , and all are learnable.

[0074] S303, projecting the object features into a scaled dot product attention space to obtain the dot product attention of the object features, and the expression is:

[0075]

[0076] where q o is a query of the object, k o is a key of the object, v o is a value of the object, ψ(·) is an object encoder, h o is a state feature, and is a transformation matrix of h o to q o , k o and v o .

[0077] In this embodiment, by flexibly using the query vector, the key vector and the value vector of the attention mechanism, the discriminability of each feature is enhanced, and the proportion of the state and the object in the original visual feature is rebalanced.

[0078] S304, according to the dot product attention of the feature of the to-be-tested original image, the dot product attention of the state feature and the dot product attention of the object feature, respectively, obtain the self-attention feature Z x of the feature of the to-be-tested original image, the self-attention feature Z s of the state feature and the self-attention feature Z o of the object feature, and the expressions are respectively:

[0079]

[0080]

[0081]

[0082] where d x is a combination proportion factor, Z x is the self-attention feature of the feature of the to-be-tested original image, d s is a first state scaling factor, Z s is the self-attention feature of the state feature, d o is a second state scaling factor, Z o is the self-attention feature of the object feature; softmax(·) is a normalization of the existing attention score, is a transpose of a matrix.

[0083] Specifically, by the above-mentioned manner, the information from different concepts (state and object) is jointly processed by using the guide model, in order to enhance the information capacity of the combined feature, the attention between the state concept, the object concept and the combined concept is calculated respectively, the query vector, the key vector and the value vector of the attention mechanism are flexibly used, and the discrimination ability of the combined feature is effectively improved, and the proportion of the state feature and the object feature in the original visual feature is rebalanced.

[0084] In an optional embodiment of the present invention, the acquisition expression x of the combined feature new for:

[0085]

[0086] in, For splicing operation.

[0087] In an optional embodiment of the present invention, the second loss function The expression is:

[0088]

[0089] Among them, c' is the combination in the state object combination pair.

[0090] In an optional embodiment of the present invention, the loss function l of the concept collaboration network is total The expression is:

[0091]

[0092]

[0093] Among them, λ t is the time parameter used to ensure the order of the training process, t is the current training cycle, m is the marginal value, and e is the base function.

[0094] Specifically, this embodiment takes into account the combination feature x new The quality of the state depends on the state characteristics h s and object characteristics h o , setting a time parameter λ t , so that the concept collaboration network can ensure the training order, that is, the state encoder and object encoder must be trained first, and then the concept interaction module is trained.

[0095] For details of the same invention, please refer to Figure 5 As shown, Figure 5 is a schematic diagram of a combined zero-shot learning device based on a concept collaboration network provided by an embodiment of the present invention. The present invention also provides a combined zero-shot learning device based on a concept collaboration network, which is applied to the combined zero-shot learning method based on a concept collaboration network provided by the above embodiment of the present application. The device includes:

[0096] The image acquisition module 201 is used to acquire the original image of the state object combination to be tested;

[0097] An image processing module 1 202 is configured to process the original image to be tested using a feature extractor of the trained concept collaboration network to obtain features of the original image to be tested;

[0098] The image processing module two 203 is configured to process the features of the to-be-tested original image using the state encoder of the trained concept collaboration network to obtain state features;

[0099] The image processing module three 204 is configured to process the features of the to-be-tested original image using the object encoder of the trained concept collaboration network to obtain object features;

[0100] The image processing module four 205 is configured to process the features of the to-be-tested original image, the state features and the object features using the concept interaction module of the trained concept collaboration network to obtain self-attention features of the features of the to-be-tested original image, self-attention features of the state features and self-attention features of the object features, respectively.

[0101] The image processing module five 206 is configured to splice the self-attention features of the features of the to-be-tested original image, the self-attention features of the state features and the self-attention features of the object features to obtain combined features.

[0102] The result obtaining module 207 is configured to learn the combined features using the learning module of the trained concept collaboration network to obtain a learning result of the to-be-tested original image.

[0103] It should be noted that, in the present document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Also, the terms “comprises”, “comprising”, or any other variant thereof are intended to cover non-exclusive inclusions, so that an item or device that comprises a list of elements does not only include those elements, but also other elements not explicitly listed. Without more limitations, an element defined by the statement “comprising a…” does not exclude the presence of additional identical elements in the item or device that includes the element. The terms “connected”, “coupled”, or similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “upper”, “lower”, “left”, “right”, and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0104] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific feature or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate different embodiments or examples described in the specification.

[0105] The above is a further detailed description of the present application in combination with specific preferred embodiments, and cannot be considered as limiting the specific implementation of the present application to these descriptions. For those skilled in the art, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, which should be considered as falling within the protection scope of the present application.

Claims

1. A combination zero-shot learning method based on a concept collaboration network, characterized in that, The method comprises the following steps: acquiring a to-be-tested original image of a state-object combination; processing the to-be-tested original image using a feature extractor of a trained concept collaboration network to obtain a feature of the to-be-tested original image; processing the feature of the to-be-tested original image using a state encoder of the trained concept collaboration network to obtain a state feature; processing the feature of the to-be-tested original image using an object encoder of the trained concept collaboration network to obtain an object feature; processing the feature of the to-be-tested original image, the state feature and the object feature using a concept interaction module of the trained concept collaboration network to respectively obtain a self-attention feature of the feature of the to-be-tested original image, a self-attention feature of the state feature and a self-attention feature of the object feature; splicing the self-attention feature of the feature of the to-be-tested original image, the self-attention feature of the state feature and the self-attention feature of the object feature to obtain a combined feature; learning the combined feature using a learning module of the trained concept collaboration network to obtain a classification result of the to-be-tested original image; the process of processing the feature of the to-be-tested original image, the state feature and the object feature using the concept interaction module of the trained concept collaboration network to respectively obtain the self-attention feature of the feature of the to-be-tested original image, the self-attention feature of the state feature and the self-attention feature of the object feature comprises the following steps: projecting the feature of the to-be-tested original image into a scaled dot-product attention space to obtain a dot-product attention of the feature of the to-be-tested original image, which is expressed as: ； wherein, is a query vector, is a key vector, is a value vector, , and are learnable matrices, is a feature of the original image to be tested. projecting the state feature into the scaled dot-product attention space to obtain a dot-product attention of the state feature, which is expressed as: ； wherein, a query for a state, a key for a state, a value for a state, a state encoder, a state feature, , and are transformation matrices from to , and ​ projecting the object feature into the scaled dot-product attention space to obtain a dot-product attention of the object feature, which is expressed as: ； wherein, is a query for an object, is a key for an object, is a value for an object, is an object encoder, is a state feature, , and is a transformation matrix from to , and . According to the dot product attention of the feature of the to-be-tested original image, the dot product attention of the state feature, and the dot product attention of the object feature, self-attention features of the feature of the to-be-tested original image, the state feature, and the object feature are respectively obtained The expressions are respectively:​​ ； ； ； wherein, is a combination scaling factor, is a self-attention feature of the characteristic of the original image to be tested, is a first state scaling factor, is a self-attention feature of the state characteristic, is a second state scaling factor, is a self-attention feature of the object characteristic; is a normalization of the existing attention score, is a transpose of a matrix.

2. The concept collaboration network based combined zero-shot learning method according to claim 1, characterized in that, The method further comprises the following steps: training a concept collaboration network; acquiring a set of states and a set of objects and generating a state-object combination pair from a state in the set of states and an object in the set of objects , expressed as: ； According to the state object combination pair, a training set is acquired The expression is: ； wherein, is an image sample, is an image in the training set, is a subset of state object combinations pairs, is the image, is the number of images in the training set; inputting the training set into the concept collaboration network, updating a loss function parameter of the concept collaboration network until a loss function value of the concept collaboration network reaches a minimum to obtain the loss function parameter of the concept collaboration network; wherein the loss function of the concept collaboration network comprises a first loss function and a second loss function, the first loss function comprises a state encoder loss function and an object encoder loss function, and the second loss function comprises a loss function of a concept interaction module; constructing the trained concept collaboration network according to the loss function parameters of the state encoder and the object encoder.

3. The concept collaboration network based combined zero-shot learning method according to claim 2, wherein, The first loss function The expression is: ； wherein, and are influence factors for balancing the influence of the loss function, is a loss function of the state encoder, is a loss function of the object encoder.

4. The concept collaboration network based combined zero-shot learning method according to claim 3, characterized in that, Loss function of the state encoder The expression is: ； Loss function of the object encoder The expression is: ； wherein, is a temperature parameter for balancing loss generated by a scaling model, is a logarithm, is an exponential function, is a state feature, is an object feature, is a state in a set of states, is an object in a set of objects.

5. The concept collaboration network based combined zero-shot learning method according to claim 1, wherein, The acquisition expression of the combination feature is is: ； wherein is a stitching operation.

6. The concept collaboration network based combined zero-shot learning method according to claim 2, characterized in that, The second loss function The expression is: ； wherein, is a combination of the pair of state objects.

7. The concept collaboration network based combined zero-shot learning method according to claim 1, characterized in that, Loss function of the concept collaboration network The expression is: ； ； wherein is a time parameter for ensuring the order of the training process, is the current training cycle, is a margin value, is a base function.

8. A combination zero-shot learning device based on a conceptual collaboration network, characterized by, The method comprises the following steps: an image acquisition module is configured to acquire a to-be-tested original image of a state-object combination; an image processing module one is configured to process the to-be-tested original image using a feature extractor of a trained concept collaboration network to obtain a feature of the to-be-tested original image; an image processing module two is configured to process the feature of the to-be-tested original image using a state encoder of the trained concept collaboration network to obtain a state feature; an image processing module three is configured to process the feature of the to-be-tested original image using an object encoder of the trained concept collaboration network to obtain an object feature; The image processing module four is configured to process the feature of the to-be-tested original image, the state feature and the object feature using the concept interaction module of the trained concept collaborative network, to obtain a self-attention feature of the feature of the to-be-tested original image, a self-attention feature of the state feature and a self-attention feature of the object feature, respectively, and the image processing module four comprises the following steps: projecting the feature of the to-be-tested original image into a scaled dot-product attention space to obtain a dot-product attention of the feature of the to-be-tested original image, and the expression is as follows: ； wherein, is a query vector, is a key vector, is a value vector, , and are learnable matrices, is a feature of the original image to be tested. projecting the state feature into the scaled dot-product attention space to obtain a dot-product attention of the state feature, and the expression is as follows: ； wherein, is a query for a state, is a key for a state, is a value for a state, is a state encoder, is a state feature, , and is a transformation matrix from to , and a transformation matrix. projecting the object feature into the scaled dot-product attention space to obtain a dot-product attention of the object feature, and the expression is as follows: ； wherein, is a query for an object, is a key for an object, is a value for an object, is an object encoder, is a state feature, , and is a transformation matrix from to , and ​ According to the dot product attention of the feature of the to-be-tested original image, the dot product attention of the state feature, and the dot product attention of the object feature, self-attention features of the feature of the to-be-tested original image, the state feature, and the object feature are respectively obtained The expressions are respectively:​​ ； ； ； wherein, is a combination scaling factor, is a self-attention feature of the characteristic of the original image to be tested, is a first state scaling factor, is a self-attention feature of the state characteristic, is a second state scaling factor, is a self-attention feature of the object characteristic; is a normalization to the existing attention score, is a transpose of a matrix; The image processing module five is configured to splice the self-attention feature of the feature of the to-be-tested original image, the self-attention feature of the state feature and the self-attention feature of the object feature to obtain a combined feature. The result obtaining module is configured to learn the combined feature using the learning module of the trained concept collaborative network to obtain a learning result of the to-be-tested original image.

Images