Instance retrieval method based on cross-semantic attention model and related device

Abstract

The application provides an instance retrieval method based on a cross-semantic attention model and related equipment. The method comprises the following steps: selecting any shot as a target shot from multiple shots, and obtaining an action prediction score vector and a location prediction score vector in the target shot; correcting the location prediction score vector and the action prediction score vector to obtain a corrected location prediction score vector and a corrected action prediction score vector; calculating an association retrieval score based on the corrected action prediction score vector, the corrected location prediction score vector, a to-be-queried action and a to-be-queried location; repeating the above steps until the multiple shots are selected to obtain multiple association retrieval scores; and obtaining a retrieval result corresponding to the to-be-queried action and the to-be-queried location based on the multiple association retrieval scores. Through the application, a shot with a semantic contradiction in a retrieval result can be effectively avoided, and the precision of "action-location" compound semantic retrieval is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to an instance retrieval method based on a cross-semantic attention model and related equipment. BACKGROUND

[0002] Compared with concept retrieval, the goal of instance retrieval is no longer to find an object of a certain class, but to find a specific individual from a large number of individuals of the class. At present, single-semantic video instance retrieval technology has made gratifying progress, but multi-concept composite semantic instance retrieval is still far from the accuracy of single-semantic instance retrieval.

[0003] The existing "action-place" composite semantic instance retrieval method directly and simply superimposes the "action" and "place" retrieval results to obtain the "action-place" binary semantic retrieval result. Since the place and the action often have a strong logical relationship, knowing the place can infer the possible action, and conversely, knowing the action can infer the place where the character is located. Therefore, this method of retrieving the action and the place in isolation without considering the semantic association information between the action and the place existing in the film and television drama can lead to retrieval results of semantically contradictory shots. SUMMARY

[0004] The main purpose of the present application is to provide an instance retrieval method based on a cross-semantic attention model and related equipment, which aims to avoid retrieval results of semantically contradictory shots and improve the accuracy of composite semantic retrieval.

[0005] In a first aspect, the present application provides an instance retrieval method based on a cross-semantic attention model, which comprises:

[0006] selecting any shot from a plurality of shots as a target shot, obtaining an action prediction score vector and a place prediction score vector in the target shot;

[0007] correcting the place prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the action prediction score vector to obtain a corrected place prediction score vector;

[0008] correcting the action prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the place prediction score vector to obtain a corrected action prediction score vector;

[0009] calculating the associated retrieval score of the query action and the query place based on the corrected action prediction score vector, the corrected place prediction score vector, the query action, and the query place;

[0010] selecting any shot from the shots that have never been selected as the target shot, returning to perform the steps of obtaining the action prediction score vector and the location prediction score vector in the target shot until all the shots are selected, obtaining a plurality of correlation retrieval scores;

[0011] obtaining a retrieval result corresponding to the action to be queried and the location to be queried based on the plurality of correlation retrieval scores.

[0012] Optionally, the step of correcting the location prediction score vector based on the attention embedding matrix, the semantic embedding matrix and the action prediction score vector to obtain a corrected location prediction score vector comprises:

[0013] substituting the action prediction score vector into a first preset formula to obtain an action attention vector and an action semantic vector, wherein the first preset formula is as follows:

[0014]

[0015] wherein k A is used to represent the action attention vector, s A is used to represent the action prediction score vector, M L is used to represent the attention embedding matrix in the action correction location module, M L ∈R |A|×A| , v A is used to represent the action semantic vector, W L is used to represent the semantic embedding matrix in the action correction location module, W L ∈R |A|×|A| , |A| is the total number of action categories, R |A|×|A| is used to represent a real number matrix with a size of A by A;

[0016] substituting the location prediction score vector and the action attention vector into a second preset formula to obtain an action-to-location attention matrix, wherein the second preset formula is as follows:

[0017]

[0018] wherein M A→L is used to represent the action-to-location attention matrix, s L is used to represent the location prediction score vector;

[0019] substituting the action semantic vector and the action-to-location attention matrix into a third preset formula to obtain a location prediction score update vector, wherein the third preset formula is as follows:

[0020]

[0021] wherein, a location prediction score update vector;

[0022] substituting the location prediction score update vector and the location prediction score vector into a fourth preset formula to obtain a corrected location prediction score vector, wherein the fourth preset formula is as follows:

[0023]

[0024] wherein, a corrected location prediction score vector.

[0025] Optionally, the step of correcting the action prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the location prediction score vector to obtain a corrected action prediction score vector comprises:

[0026] substituting the location prediction score vector into a fifth preset formula to obtain an action attention vector and an action semantic vector, wherein the fifth preset formula is as follows:

[0027]

[0028] wherein, k L a location attention vector, s L a location prediction score vector, M L′ an attention embedding matrix in the location correction action module, M L′ ∈R |L|×|L| , v L a location semantic vector, W L′ a semantic embedding matrix in the location correction action module, W L′ ∈R |L|×|L| |L| denotes a total number of location categories, R |L|×|L| denotes a real matrix with a size of L by L;

[0029] substituting the action prediction score vector and the location attention vector into a sixth preset formula to obtain a location-to-action attention matrix, wherein the sixth preset formula is as follows:

[0030]

[0031] wherein, M L→A a location-to-action attention matrix, s A an action prediction score vector;

[0032] substituting the location semantic vector and the location-to-action attention matrix into a seventh preset formula to obtain an action prediction score update vector, wherein the seventh preset formula is as follows:

[0033]

[0034] wherein, is used to represent the action prediction score update vector;

[0035] substituting the action prediction score update vector and the action prediction score vector into an eighth preset formula to obtain a modified location prediction score vector, wherein the eighth preset formula is as follows:

[0036]

[0037] wherein, is used to represent the modified action prediction score vector.

[0038] Optionally, the step of calculating the association retrieval score of the query action and the query location based on the modified action prediction score vector, the modified location prediction score vector, the query action, and the query location comprises:

[0039] selecting a first prediction score sub-vector corresponding to the query action from the modified action prediction score vector;

[0040] selecting a second prediction score sub-vector corresponding to the query location from the modified location prediction score vector;

[0041] calculating the product of the first prediction score sub-vector and the second prediction score sub-vector, and taking the product as the association retrieval score of the query action and the query location.

[0042] Optionally, the step of obtaining the retrieval result corresponding to the query action and the query location based on the plurality of association retrieval scores comprises:

[0043] performing a max-pooling operation on the plurality of association retrieval scores to obtain the retrieval result corresponding to the query action and the query location.

[0044] Optionally, before the step of modifying the location prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the action prediction score vector to obtain the modified location prediction score vector, the method further comprises:

[0045] inputting the action prediction score vector and the location prediction score vector in the training set into the location modification action module to obtain the modified action prediction score vector output by the location modification action module;

[0046] inputting the action prediction score vector and the location prediction score vector in the training set into the action modification location module to obtain the modified location prediction score vector output by the action modification location module;

[0047] a first loss between the modified action prediction score vector output by the action modification location module and a preset action score vector;

[0048] a second loss between the modified location prediction score vector output by the action modification location module and a preset location score vector;

[0049] a third loss obtained by substituting the first loss and the second loss into a ninth preset formula, wherein the ninth preset formula is as follows:

[0050]

[0051] a first loss between the modified action prediction score vector output by the action modification location module and a preset action score vector, a second loss between the modified location prediction score vector output by the action modification location module and a preset location score vector, and λ represents a loss fusion weight, a modified action prediction score vector, a modified location prediction score vector, a preset action score vector, a preset location score vector, a third loss;

[0052] a gradient of the third loss is calculated, and the gradient is back propagated to update parameters of the attention embedding matrix and parameters of the semantic embedding matrix in the action modification location module and parameters of the attention embedding matrix and parameters of the semantic embedding matrix in the action modification location module;

[0053] the step of inputting the action prediction score vector and the location prediction score vector in the training set into the action modification location module to obtain the modified action prediction score vector output by the action modification location module is performed again with a new action prediction score vector in the training set as the action prediction score vector and a new location prediction score vector in the training set as the location prediction score vector;

[0054] when the number of cycles reaches a preset number, the trained attention embedding matrix and the trained semantic embedding matrix in the action modification location module and the trained attention embedding matrix and the trained semantic embedding matrix in the action modification location module are obtained.

[0055] In a second aspect, the present application further provides a case retrieval device based on a cross-semantic attention model, which comprises:

[0056] a prediction score vector obtaining module, configured to select any one of the plurality of shots as a target shot, and obtain an action prediction score vector and a location prediction score vector in the target shot;

[0057] a first correction module, configured to correct the location prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the action prediction score vector, to obtain a corrected location prediction score vector;

[0058] a second correction module, configured to correct the action prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the location prediction score vector, to obtain a corrected action prediction score vector;

[0059] a calculation module, configured to calculate an association retrieval score of the to-be-queried action and the to-be-queried location based on the corrected action prediction score vector, the corrected location prediction score vector, the to-be-queried action, and the to-be-queried location;

[0060] a loop module, configured to select any one of the shots that have not been selected as a target shot, and return to execute the step of obtaining the action prediction score vector and the location prediction score vector in the target shot, until all the plurality of shots are selected, to obtain a plurality of association retrieval scores;

[0061] a retrieval result obtaining module, configured to obtain a retrieval result corresponding to the to-be-queried action and the to-be-queried location based on the plurality of association retrieval scores.

[0062] Optionally, the first correction module is configured to:

[0063] substitute the action prediction score vector into a first preset formula to calculate an action attention vector and an action semantic vector, wherein the first preset formula is as follows:

[0064]

[0065] wherein k A is used to represent the action attention vector, s A is used to represent the action prediction score vector, M L is used to represent the attention embedding matrix in the action correction location module, M L ∈R |A|×A| , v A is used to represent the action semantic vector, W L is used to represent the semantic embedding matrix in the action correction location module, W L ∈R |A|×|A| , |A| is a total number of action categories, R |A|×|A| is used to represent a real matrix with a size of A by A;

[0066] The action-to-place attention matrix is calculated by substituting the place prediction score vector and the action attention vector into a second preset formula, wherein the second preset formula is as follows:

[0067]

[0068] wherein M A→L is used to represent the action-to-place attention matrix, s L is used to represent the place prediction score vector.

[0069] The place prediction score update vector is calculated by substituting the action semantic vector and the action-to-place attention matrix into a third preset formula, wherein the third preset formula is as follows:

[0070]

[0071] wherein, is used to represent the place prediction score update vector.

[0072] The corrected place prediction score vector is calculated by substituting the place prediction score update vector and the place prediction score vector into a fourth preset formula, wherein the fourth preset formula is as follows:

[0073]

[0074] wherein, is used to represent the corrected place prediction score vector.

[0075] In a third aspect, the present application further provides a cross-semantic attention model-based instance retrieval device, which comprises a processor, a memory, and a cross-semantic attention model-based instance retrieval program stored in the memory and executable by the processor, wherein the cross-semantic attention model-based instance retrieval program, when executed by the processor, implements the steps of the cross-semantic attention model-based instance retrieval method as described above.

[0076] In a fourth aspect, the present application further provides a readable storage medium, which stores a cross-semantic attention model-based instance retrieval program, wherein the cross-semantic attention model-based instance retrieval program, when executed by a processor, implements the steps of the cross-semantic attention model-based instance retrieval method as described above.

[0077] In the present application, any one of the multiple shots is selected as a target shot, the action prediction score vector and the location prediction score vector in the target shot are obtained, the location prediction score vector is corrected based on the attention embedding matrix, the semantic embedding matrix and the action prediction score vector, the corrected location prediction score vector is obtained, the action prediction score vector is corrected based on the attention embedding matrix, the semantic embedding matrix and the location prediction score vector, the corrected action prediction score vector is obtained, the correlation retrieval score of the query action and the query location is calculated based on the corrected action prediction score vector, the corrected location prediction score vector, the query action and the query location, any one of the shots that has not been selected is selected as a target shot, the step of obtaining the action prediction score vector and the location prediction score vector in the target shot is returned to execute until all the multiple shots are selected, and the multiple correlation retrieval scores are obtained. Based on the multiple correlation retrieval scores, the retrieval result corresponding to the query action and the query location is obtained. Through the present application, the effective composite semantic retrieval method is used from the visual rather than the script way, the semantic association information of the action semantic and the location semantic in the image features in the video of the film and television drama is fully utilized to solve the semantic contradiction problem in the "action-location" composite semantic retrieval, the semantic contradiction shot in the retrieval result is effectively avoided, and the precision of the "action-location" composite semantic retrieval is improved. BRIEF DESCRIPTION OF DRAWINGS

[0078] Figure 1 The flowchart of the first embodiment of the example retrieval method based on the cross-semantic attention model of the present application is shown.

[0079] Figure 2 The flowchart of the first embodiment of the example retrieval method based on the cross-semantic attention model of the present application is shown.

[0080] Figure 3 The flowchart of the second embodiment of the example retrieval method based on the cross-semantic attention model of the present application is shown.

[0081] Figure 4 The flowchart of the second embodiment of the example retrieval method based on the cross-semantic attention model of the present application is shown. Figure 1 The flowchart of the second embodiment of the example retrieval method based on the cross-semantic attention model of the present application is shown.

[0082] Figure 5 The functional module diagram of the first embodiment of the example retrieval device based on the cross-semantic attention model of the present application is shown.

[0083] Figure 6 The hardware structure diagram of the example retrieval device based on the cross-semantic attention model involved in the embodiment of the present application is shown.

[0084] The implementation of the present application, the functional features and the advantages will be further described with reference to the drawings. DETAILED DESCRIPTION

[0085] It is to be understood that the specific embodiments described herein are merely illustrative of the present application and should not be used to limit the scope of the present application.

[0086] In a first aspect, the embodiments of the present application provide a cross-semantic attention model-based instance retrieval method.

[0087] In an embodiment, the cross-semantic attention model-based instance retrieval method comprises the following steps: Figure 1 Figure 1 FIG. 1 is a flowchart of a first embodiment of the cross-semantic attention model-based instance retrieval method according to the present application. As shown in FIG. 1, the cross-semantic attention model-based instance retrieval method comprises the following steps: Figure 1

[0088] In step S10, any shot is selected as a target shot from a plurality of shots, and an action prediction score vector and a location prediction score vector in the target shot are obtained.

[0089] In the embodiment, a video structuring tool is used to cut a film and television video into shots, any shot is selected as a target shot from a plurality of shots, key frames are obtained by uniformly sampling in the target shot video, an ACAM algorithm and a PPDM algorithm are used to detect the action type on the key frame image, a ResNet is used to detect the scene type of the key frame, and the score vectors on the key frames belonging to the same shot are averaged to obtain the action prediction score vector and the location prediction score vector of the shot. For dynamic actions, an actor conditioned attention maps (ACAM) based on the action is adopted. The ACAM algorithm uses an attention module to sort the relevance of each spatio-temporal region to the detected behavior body, is suitable for observing complex events with multiple behavior bodies, and is a near real-time algorithm. For static actions, a parallel point detection and matching (PPDM) algorithm for real-time character interaction is selected. The PPDM is a single-stage real-time algorithm based on the anchor-free detection idea, and is more suitable for large-scale data sets. The feature extraction of the scene retrieval module extracts 1024-dimensional semantic-aware scene features through a ResNet-18 structure. After extracting the scene features, the cosine similarity is used to measure the feature similarity, and the scene category and the corresponding score on each key frame are calculated.

[0090] In step S20, the location prediction score vector is corrected based on the attention embedding matrix, the semantic embedding matrix, and the action prediction score vector to obtain a corrected location prediction score vector.

[0091] ​​In this embodiment, the location prediction score vector is corrected based on the action prediction score vector, the attention embedding matrix in the action correction location module and the semantic embedding matrix in the action correction location module using the cross-semantic attention mechanism to obtain a corrected location prediction score vector. The attention embedding matrix and the semantic embedding matrix in the action correction location module are both trained attention embedding matrix and semantic embedding matrix.

[0092] Further, in an embodiment, step S20 comprises:

[0093] The action prediction score vector is substituted into a first preset formula to calculate an action attention vector and an action semantic vector, wherein the first preset formula is as follows:

[0094]

[0095] wherein k A is used to represent the action attention vector, s A is used to represent the action prediction score vector, M L is used to represent the attention embedding matrix in the action correction location module, M L ∈R |A|×A| , v A is used to represent the action semantic vector, W L is used to represent the semantic embedding matrix in the action correction location module, W L ∈R |A|×|A| , |A| is the total number of action categories, R |A|×A| is used to represent a real matrix with a size of A by A;

[0096] The location prediction score vector and the action attention vector are substituted into a second preset formula to calculate an action-to-location attention matrix, wherein the second preset formula is as follows:

[0097]

[0098] wherein M A→L is used to represent the action-to-location attention matrix, s L is used to represent the location prediction score vector;

[0099] The action semantic vector and the action-to-location attention matrix are substituted into a third preset formula to calculate a location prediction score update vector, wherein the third preset formula is as follows:

[0100]

[0101] wherein is used to represent the location prediction score update vector;

[0102] The place prediction score update vector and the place prediction score vector are substituted into a fourth preset formula to obtain a corrected place prediction score vector, wherein the fourth preset formula is as follows:

[0103]

[0104] wherein, is used to represent the corrected place prediction score vector.

[0105] In the embodiment, it is assumed that the action prediction score vector of the lens at the initial time is S A ∈R |A| , and the place prediction score vector is S L ∈R |L| The attention embedding matrix and the semantic embedding matrix in the action correction place module are introduced to correct the place prediction score vector by using the action prediction score vector. Specifically, the action prediction score vector is substituted into a first preset formula to obtain an action attention vector and an action semantic vector, wherein the first preset formula is as follows:

[0106]

[0107] k A is used to represent the action attention vector, s A is used to represent the action prediction score vector, M L is used to represent the attention embedding matrix in the action correction place module, M L ∈R |A|×|A| , v A is used to represent the action semantic vector, W L is used to represent the semantic embedding matrix in the action correction place module, W L ∈R |A|×|A| , |A| is the total number of action categories, R |A|×|A| is used to represent a real matrix with a size of A by A.

[0108] The place prediction score vector and the action attention vector are substituted into a second preset formula to obtain an action-to-place attention matrix, wherein the second preset formula is as follows:

[0109]

[0110] wherein, M A→L is used to represent the action-to-place attention matrix, s L is used to represent the place prediction score vector.

[0111] The action semantic vector and the action-to-place attention matrix are substituted into a third preset formula to obtain a place prediction score update vector, wherein the third preset formula is as follows:

[0112]

[0113] wherein, is used to represent the location prediction score update vector.

[0114] The location prediction score update vector and the location prediction score vector are substituted into a fourth preset formula to obtain a corrected location prediction score vector, wherein the fourth preset formula is as follows:

[0115]

[0116] wherein, is used to represent the corrected location prediction score vector.

[0117] In step S30, the action prediction score vector is corrected based on the attention embedding matrix, the semantic embedding matrix, and the location prediction score vector to obtain a corrected action prediction score vector.

[0118] In this embodiment, the attention embedding matrix and the semantic embedding matrix in the location correction action module are introduced, and the action prediction score vector is corrected by using the location prediction score vector to obtain the corrected action prediction score vector. The attention embedding matrix and the semantic embedding matrix in the location correction action module are both trained attention embedding matrices and semantic embedding matrices.

[0119] Further, in an embodiment, step S30 comprises:

[0120] The location prediction score vector is substituted into a fifth preset formula to obtain a location attention vector and a location semantic vector, wherein the fifth preset formula is as follows:

[0121]

[0122] wherein, k L is used to represent the location attention vector, s L is used to represent the location prediction score vector, M L′ is used to represent the attention embedding matrix in the location correction action module, M L′ ∈R |L|×|L| , v L is used to represent the location semantic vector, W L′ is used to represent the semantic embedding matrix in the location correction action module, W L′ ∈R |L|×|L| , |L| is used to represent the total number of location categories, R |L|×|L| is used to represent a real number matrix with a size of L by L.

[0123] The action prediction score vector and the location attention vector are substituted into a sixth preset formula to obtain a location-to-action attention matrix, wherein the sixth preset formula is as follows:

[0124]

[0125] wherein M L→A is used to represent the location-to-action attention matrix, s A is used to represent the action prediction score vector.

[0126] The location semantic vector and the location-to-action attention matrix are substituted into a seventh preset formula to obtain an action prediction score update vector, wherein the seventh preset formula is as follows:

[0127]

[0128] wherein, is used to represent the action prediction score update vector.

[0129] The action prediction score update vector and the action prediction score vector are substituted into an eighth preset formula to obtain a corrected location prediction score vector, wherein the eighth preset formula is as follows:

[0130]

[0131] wherein, is used to represent the corrected action prediction score vector.

[0132] In the embodiment, with reference to Figure 2 , Figure 2 is a flowchart of a first embodiment of the cross-semantic attention model. As shown in Figure 2 , it is assumed that the action prediction score vector of the shot at the initial time is s A ∈R |A| , and the location prediction score vector is s L ∈R |L| . The location prediction score vector s L is substituted into a fifth preset formula to obtain a location attention vector and a location semantic vector, wherein the fifth preset formula is as follows:

[0133]

[0134] wherein k L is used to represent the location attention vector, s L is used to represent the location prediction score vector, M L′ is used to represent an attention embedding matrix in the location correction action module, M L′ ∈R |L|×|L| , and v LW is used to represent location semantic vectors. L′ W is used to represent the semantic embedding matrix in the location correction action module. L′ ∈R |L|×|L| |L| is used to represent the total number of location categories, R |L|×|L| Used to represent a real matrix of size L by L.

[0135] Substituting the action prediction score vector and the location attention vector into the sixth preset formula, the location-to-action attention matrix is ​​calculated. The sixth preset formula is as follows:

[0136]

[0137] Among them, M L→A The attention matrix s is used to represent the location-to-action process. A Used to represent the action prediction score vector.

[0138] Substituting the location semantic vector and the location-to-action attention matrix into the seventh preset formula, the action prediction score update vector is calculated. The seventh preset formula is as follows:

[0139]

[0140] in, Used to represent the action prediction score update vector.

[0141] Substituting the motion prediction score update vector and the motion prediction score vector into the eighth preset formula, the corrected location prediction score vector is calculated. The eighth preset formula is as follows:

[0142]

[0143] in, Used to represent the corrected action prediction score vector.

[0144] Step S40: Calculate the association retrieval score between the queried action and the queried location based on the corrected action prediction score vector, the corrected location prediction score vector, the queried action, and the queried location.

[0145] In this embodiment, refer to Figure 3 , Figure 3 This is a flowchart illustrating a second embodiment of the cross-semantic attention model of the present invention. Figure 3 As shown, the location prediction score vector s is obtained through a cross-semantic attention mechanism. L After making corrections, the corrected location prediction score vector is obtained. The action prediction score vector s is modified by a cross-semantic attention mechanism A The modified action prediction score vector s is obtained by modifying the action prediction score vector s The modified action prediction score vector s is obtained by modifying the action prediction score vector s The modified location prediction score vector s is obtained by modifying the location prediction score vector s The target shot is selected from the shots, and the action query a and the location query t are used to calculate the relevance search score of the target shot with respect to the binary semantic query (a, t).

[0146] Further, in an embodiment, the relevance search score of the target shot with respect to the binary semantic query (a, t) is calculated by Figure 4 , Figure 4 Figure 1 The flowchart of step S40 in the embodiment is shown in FIG. 4. As shown in FIG. 4, step S40 includes the following steps. Figure 4

[0147] Step S401: A first prediction score sub-vector corresponding to the action query a is selected from the modified action prediction score vector s.

[0148] Step S402: A second prediction score sub-vector corresponding to the location query t is selected from the modified location prediction score vector s.

[0149] Step S403: The product of the first prediction score sub-vector and the second prediction score sub-vector is calculated, and the product is used as the relevance search score of the action query a and the location query t.

[0150] In the embodiment, the first prediction score sub-vector corresponding to the action query a is selected from the modified action prediction score vector s The second prediction score sub-vector corresponding to the location query t is selected from the modified location prediction score vector s

[0151] The product of the first prediction score sub-vector and the second prediction score sub-vector is calculated, and the product is used as the relevance search score of the action query a and the location query t.

[0152] Step S50: Any shot is selected from the shots that have not been selected as the target shot, and the step of obtaining the action prediction score vector and the location prediction score vector in the target shot is returned until all the shots are selected, and a plurality of relevance search scores are obtained.

[0153] ​​​​​​​In this embodiment, any lens that has not been selected is reselected as the target lens again, and the steps of obtaining the action prediction score vector and the location prediction score vector in the target lens are returned to be executed, that is, the steps S10 to S50 are returned to be executed until all the lenses are selected, and the plurality of correlation search scores are obtained.

[0154] In step S60, the search result corresponding to the action to be queried and the location to be queried is obtained based on the plurality of correlation search scores.

[0155] In this embodiment, the pooling operation is performed based on the plurality of correlation search scores, and the search result corresponding to the action to be queried and the location to be queried is obtained.

[0156] Further, in an embodiment, the step S60 comprises:

[0157] The maximum pooling operation is performed on the plurality of correlation search scores, and the search result corresponding to the action to be queried and the location to be queried is obtained.

[0158] In this embodiment, the maximum pooling operation is performed on the plurality of correlation search scores, and the maximum value in the plurality of correlation search scores is obtained, which is the search result corresponding to the action to be queried and the location to be queried.

[0159] In this embodiment, any lens is selected as the target lens from the plurality of lenses, the action prediction score vector and the location prediction score vector in the target lens are obtained, the location prediction score vector is corrected based on the attention embedding matrix, the semantic embedding matrix and the action prediction score vector to obtain the corrected location prediction score vector, the action prediction score vector is corrected based on the attention embedding matrix, the semantic embedding matrix and the location prediction score vector to obtain the corrected action prediction score vector, the correlation search score of the action to be queried and the location to be queried is calculated based on the corrected action prediction score vector, the corrected location prediction score vector, the action to be queried and the location to be queried, any lens is selected as the target lens from the lenses that have not been selected, and the step of obtaining the action prediction score vector and the location prediction score vector in the target lens is returned to be executed until all the lenses are selected, the plurality of correlation search scores are obtained, and the search result corresponding to the action to be queried and the location to be queried is obtained based on the plurality of correlation search scores. Through this embodiment, from the visual rather than the script approach, an effective composite semantic search method is used, the semantic association information of the action semantic and the location semantic in the image features in the video of the film and television drama is fully utilized to solve the problem of semantic contradiction in the "action-location" composite semantic search, the lens with semantic contradiction in the search result is effectively avoided, and the precision of the "action-location" composite semantic search is improved.

[0160] Further, in an embodiment, before the step S20, the following steps are included:

[0161] inputting the action prediction score vector and the location prediction score vector in the training set into the location correction action module to obtain a corrected action prediction score vector output by the location correction action module;

[0162] inputting the action prediction score vector and the location prediction score vector in the training set into the action correction location module to obtain a corrected location prediction score vector output by the action correction location module;

[0163] calculating a first loss between the corrected action prediction score vector output by the location correction action module and a preset action score vector;

[0164] calculating a second loss between the corrected location prediction score vector output by the action correction location module and a preset location score vector;

[0165] substituting the first loss and the second loss into a ninth preset formula to obtain a third loss, wherein the ninth preset formula is as follows:

[0166]

[0167] the first loss between the corrected action prediction score vector and the preset action score vector in the location correction action module, the second loss between the corrected location prediction score vector and the preset location score vector in the action correction location module, and λ represents a loss fusion weight, the corrected action prediction score vector, the corrected location prediction score vector, the preset action score vector, the preset location score vector;

[0168] calculating the gradient of the third loss, and updating the parameters of the attention embedding matrix and the parameters of the semantic embedding matrix in the location correction action module and the parameters of the attention embedding matrix and the parameters of the semantic embedding matrix in the action correction location module through back propagation of the gradient;

[0169] using a new action prediction score vector in the training set as the action prediction score vector and using a new location prediction score vector in the training set as the location prediction score vector, and returning to execute the step of inputting the action prediction score vector and the location prediction score vector in the training set into the location correction action module to obtain a corrected action prediction score vector output by the location correction action module;

[0170] When the number of cycles reaches the preset number, the attention embedding matrix and the semantic embedding matrix trained in the place correction action module and the attention embedding matrix and the semantic embedding matrix trained in the action correction place module are obtained.

[0171] In the embodiment, a cross-semantic attention model is used, which has the characteristics of few parameters and easy training, has low requirements on computer computing power, and has strong deployability. The cross-semantic attention model includes a place correction action module and an action correction place module. The action prediction score vector and the place prediction score vector in the training set are input into the place correction action module in the cross-semantic attention model, and through the attention embedding matrix and the semantic embedding matrix in the place correction action module, a corrected action prediction score vector output by the place correction action module is obtained. The action prediction score vector and the place prediction score vector in the training set are input into the action correction place module, and through the attention embedding matrix and the semantic embedding matrix in the action correction place module, a corrected place prediction score vector output by the action correction place module is obtained.

[0172] The first loss between the corrected action prediction score vector output by the place correction action module and the preset action score vector is calculated

[0173] The second loss between the corrected place prediction score vector output by the action correction place module and the preset place score vector is calculated

[0174] The first loss and the second loss are substituted into the ninth preset formula to obtain a third loss, wherein the ninth preset formula is as follows:

[0175]

[0176] for representing the first loss between the corrected action prediction score vector and the preset action score vector in the place correction action module, for representing the second loss between the corrected place prediction score vector and the preset place score vector in the action correction place module, and λ is used to represent a loss fusion weight, for representing the corrected action prediction score vector, for representing the corrected place prediction score vector, for representing the preset action score vector, for representing the preset place score vector, for representing the third loss.

[0177] The gradient of the third loss is calculated, and the gradient is back-propagated to update the parameters of the attention embedding matrix and the parameters of the semantic embedding matrix in the action modification place correction module, and the parameters of the attention embedding matrix and the parameters of the semantic embedding matrix in the place modification action module.

[0178] The step of inputting the action prediction score vector and the place prediction score vector in the training set into the place correction action module to obtain the corrected action prediction score vector output by the place correction action module is returned, that is, the place correction action module and the action correction place module in the cross-semantic attention model are retrained, with the new action prediction score vector in the training set as the action prediction score vector, and the new place prediction score vector in the training set as the place prediction score vector.

[0179] When the number of cycles, that is, the number of training times, reaches a preset number, the trained attention embedding matrix and the trained semantic embedding matrix in the place correction action module, and the trained attention embedding matrix and the trained semantic embedding matrix in the action correction place module are obtained.

[0180] In a second aspect, an embodiment of the present application also provides an instance retrieval device based on a cross-semantic attention model.

[0181] In an embodiment, the instance retrieval device based on the cross-semantic attention model comprises: Figure 5 , Figure 5 FIG. 1 is a schematic diagram of functional modules of an embodiment of the instance retrieval device based on the cross-semantic attention model of the present application. As shown in FIG. 1, the instance retrieval device based on the cross-semantic attention model comprises: Figure 5

[0182] The prediction score vector acquisition module 10 is configured to select any one of the plurality of shots as a target shot, and acquire an action prediction score vector and a place prediction score vector in the target shot.

[0183] The first correction module 20 is configured to correct the place prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the action prediction score vector, to obtain a corrected place prediction score vector.

[0184] The second correction module 30 is configured to correct the action prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the place prediction score vector, to obtain a corrected action prediction score vector.

[0185] The calculation module 40 is configured to calculate an associated retrieval score of the query action and the query place based on the corrected action prediction score vector, the corrected place prediction score vector, the query action, and the query place.

[0186] ​The circulation module 50 is configured to select any shot from the shots that have not been selected as the target shot, return to execute the steps of obtaining the action prediction score vector and the location prediction score vector in the target shot until all the shots are selected, and obtain a plurality of association search scores.

[0187] The search result obtaining module 60 is configured to obtain search results corresponding to the to-be-searched action and the to-be-searched location based on the plurality of association search scores.

[0188] Further, in an embodiment, the first correction module 20 is configured to:

[0189] The action prediction score vector is substituted into a first preset formula to calculate an action attention vector and an action semantic vector, wherein the first preset formula is as follows:

[0190]

[0191] wherein k A is used to represent the action attention vector, s A is used to represent the action prediction score vector, M L is used to represent an attention embedding matrix in the action correction location module, M L ∈R |A|×A| , v A is used to represent the action semantic vector, W L is used to represent a semantic embedding matrix in the action correction location module, W L ∈R |A|×|A| , |A| is the total number of action categories, R |A|×|A| is used to represent a real number matrix with a size of A by A;

[0192] The location prediction score vector and the action attention vector are substituted into a second preset formula to calculate an action-to-location attention matrix, wherein the second preset formula is as follows:

[0193]

[0194] wherein M A→L is used to represent the action-to-location attention matrix, s L is used to represent the location prediction score vector;

[0195] The action semantic vector and the action-to-location attention matrix are substituted into a third preset formula to calculate a location prediction score update vector, wherein the third preset formula is as follows:

[0196]

[0197] wherein is used to represent the location prediction score update vector;

[0198] The fourth preset formula is as follows:

[0199]

[0200] wherein, is used to represent the modified place prediction score vector.

[0201] Further, in an embodiment, the second correction module 30 is configured to:

[0202] The fifth preset formula is as follows:

[0203]

[0204] wherein, k L is used to represent the place attention vector, s L is used to represent the place prediction score vector, M L′ is used to represent the attention embedding matrix in the place correction action module, M L′ ∈R |L|×|L| , v L is used to represent the place semantic vector, W L′ is used to represent the semantic embedding matrix in the place correction action module, W L′ ∈R |L|×|L| , |L| is used to represent the total number of place categories, R |L|×|L| is used to represent a real number matrix with a size of L by L;

[0205] The sixth preset formula is as follows:

[0206]

[0207] wherein, M L→A is used to represent the place-to-action attention matrix, s A is used to represent the action prediction score vector;

[0208] The seventh preset formula is as follows:

[0209]

[0210] wherein, is used to represent the action prediction score update vector;

[0211] The action prediction score update vector and the action prediction score vector are substituted into an eighth preset formula to obtain a modified location prediction score vector, wherein the eighth preset formula is as follows:

[0212]

[0213] wherein, is used to represent the modified action prediction score vector.

[0214] Further, in an embodiment, the calculation module 40 is configured to:

[0215] select a first prediction score sub-vector corresponding to the to-be-queried action from the modified action prediction score vector;

[0216] select a second prediction score sub-vector corresponding to the to-be-queried location from the modified location prediction score vector;

[0217] calculate a product of the first prediction score sub-vector and the second prediction score sub-vector, and use the product as an association retrieval score of the to-be-queried action and the to-be-queried location.

[0218] Further, in an embodiment, the retrieval result acquisition module is configured to:

[0219] perform a max-pooling operation on the plurality of association retrieval scores to obtain a retrieval result corresponding to the to-be-queried action and the to-be-queried location.

[0220] Further, in an embodiment, the cycle module 50 is further configured to:

[0221] input the action prediction score vector and the location prediction score vector in the training set into the location modification action module to obtain a modified action prediction score vector output by the location modification action module;

[0222] input the action prediction score vector and the location prediction score vector in the training set into the action modification location module to obtain a modified location prediction score vector output by the action modification location module;

[0223] calculate a first loss between the modified action prediction score vector output by the location modification action module and a preset action score vector;

[0224] calculate a second loss between the modified location prediction score vector output by the action modification location module and a preset location score vector;

[0225] substitute the first loss and the second loss into a ninth preset formula to obtain a third loss, wherein the ninth preset formula is as follows:

[0226]

[0227] a first loss for representing a difference between the modified action prediction score vector and a preset action score vector in the location modification action module, a second loss for representing a difference between the modified location prediction score vector and a preset location score vector in the action modification location module, and λ represents a loss fusion weight, for representing the modified action prediction score vector, for representing the modified location prediction score vector, for representing the preset action score vector, for representing the preset location score vector, for representing the third loss;

[0228] calculating a gradient of the third loss, and back-propagating the gradient to update parameters of the attention embedding matrix and the semantic embedding matrix in the location modification action module and the action modification location module;

[0229] returning to execute the step of inputting the action prediction score vector and the location prediction score vector in the training set into the location modification action module to obtain the modified action prediction score vector output by the location modification action module, with the new action prediction score vector in the training set as the action prediction score vector and the new location prediction score vector in the training set as the location prediction score vector;

[0230] when the number of iterations reaches a preset number, obtaining the trained attention embedding matrix and the semantic embedding matrix in the location modification action module and the trained attention embedding matrix and the semantic embedding matrix in the action modification location module.

[0231] Corresponding to each step in the above embodiment of the instance retrieval method based on the cross-semantic attention model, the functions of each module in the above instance retrieval device based on the cross-semantic attention model are implemented, and the functions and implementation processes are not repeated here.

[0232] In a third aspect, an embodiment of the present application provides an instance retrieval device based on a cross-semantic attention model. The instance retrieval device based on the cross-semantic attention model can be a personal computer (PC), a notebook computer, a server, or other devices with data processing functions.

[0233] Reference Figure 6 , Figure 6This is a schematic diagram of the hardware structure of the instance retrieval device based on a cross-semantic attention model involved in the embodiments of the present invention. In this embodiment, the instance retrieval device based on a cross-semantic attention model may include a processor 1001 (e.g., a Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize communication between these components; the user interface 1003 may include a display screen or an input unit such as a keyboard; the network interface 1004 may optionally include a standard wired interface or a wireless interface (e.g., Wireless Fidelity, Wi-Fi interface); the memory 1005 may be high-speed random access memory (RAM) or stable memory (non-volatile memory), such as a disk storage device; the memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001. Those skilled in the art will understand that… Figure 6 The hardware structure shown does not constitute a limitation of the invention and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0234] Continue to refer to Figure 6 , Figure 6 The memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an instance retrieval program based on a cross-semantic attention model. The processor 1001 can call the instance retrieval program based on the cross-semantic attention model stored in the memory 1005 and execute the instance retrieval method based on the cross-semantic attention model provided in this embodiment of the invention.

[0235] Fourthly, embodiments of the present invention also provide a readable storage medium.

[0236] The present invention stores an instance retrieval program based on a cross-semantic attention model on a readable storage medium, wherein when the instance retrieval program based on the cross-semantic attention model is executed by a processor, it implements the steps of the instance retrieval method based on the cross-semantic attention model as described above.

[0237] The method implemented when the instance retrieval program based on the cross-semantic attention model is executed can be referred to in various embodiments of the instance retrieval method based on the cross-semantic attention model of the present invention, and will not be repeated here.

[0238] It should be noted that, in this document, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0239] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0240] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and the necessary general hardware platform, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) as described above, and includes a number of instructions for causing a terminal device to execute the methods described in the various embodiments of the present application.

[0241] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation made by using the content of the specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for instance retrieval based on cross-semantic attention model, characterized in that, The instance retrieval method based on the cross-semantic attention model comprises: selecting any shot as a target shot from the multiple shots, and obtaining an action prediction score vector and a location prediction score vector in the target shot; correcting the location prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the action prediction score vector to obtain a corrected location prediction score vector; correcting the action prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the location prediction score vector to obtain a corrected action prediction score vector; calculating an associated retrieval score of the query action and the query location based on the corrected action prediction score vector, the corrected location prediction score vector, the query action, and the query location; selecting any shot as a target shot from the shots that have not been selected, and returning to the step of obtaining the action prediction score vector and the location prediction score vector in the target shot until all the shots are selected, thereby obtaining multiple associated retrieval scores; obtaining a retrieval result corresponding to the query action and the query location based on the multiple associated retrieval scores; the step of correcting the location prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the action prediction score vector to obtain a corrected location prediction score vector comprises: substituting the action prediction score vector into a first preset formula to calculate an action attention vector and an action semantic vector, wherein the first preset formula is as follows: wherein, for representing an action attention vector, for representing an action prediction score vector, for representing an attention embedding matrix in the action correction location module, , for representing an action semantic vector, for representing a semantic embedding matrix in the action correction location module, , is the total number of action classes, for representing a real-valued matrix of size A by A; substituting the location prediction score vector and the action attention vector into a second preset formula to calculate an action-to-location attention matrix, wherein the second preset formula is as follows: wherein, an attention matrix for representing actions to places, a place prediction score vector; substituting the action semantic vector and the action-to-location attention matrix into a third preset formula to calculate a location prediction score update vector, wherein the third preset formula is as follows: wherein, for representing a location prediction score update vector; substituting the location prediction score update vector and the location prediction score vector into a fourth preset formula to calculate the corrected location prediction score vector, wherein the fourth preset formula is as follows: wherein, for representing the modified location prediction score vector; the step of correcting the action prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the location prediction score vector to obtain a corrected action prediction score vector comprises: substituting the location prediction score vector into a fifth preset formula to calculate an action attention vector and an action semantic vector, wherein the fifth preset formula is as follows: wherein, for representing a location attention vector, for representing a location prediction score vector, for representing an attention embedding matrix in the location revision action module, , for representing a location semantic vector, for representing a semantic embedding matrix in the location revision action module, , for representing a total number of location categories, for representing a real number matrix of size L by L; substituting the action prediction score vector and the location attention vector into a sixth preset formula to calculate a location-to-action attention matrix, wherein the sixth preset formula is as follows: wherein, a location-to-action attention matrix, a vector of action prediction scores; substituting the location semantic vector and the location-to-action attention matrix into a seventh preset formula to calculate an action prediction score update vector, wherein the seventh preset formula is as follows: wherein, for representing an action prediction score update vector; substituting the action prediction score update vector and the action prediction score vector into an eighth preset formula to calculate the corrected location prediction score vector, wherein the eighth preset formula is as follows: wherein, for representing the modified action prediction score vector. 2.The instance retrieval method based on cross-semantic attention model according to claim 1, wherein, the step of calculating the associated retrieval score of the query action and the query location based on the corrected action prediction score vector, the corrected location prediction score vector, the query action, and the query location comprises: selecting a first prediction score sub-vector corresponding to the dimension of the to-be-queried action from the modified action prediction score vector; selecting a second prediction score sub-vector corresponding to the dimension of the to-be-queried location from the modified location prediction score vector; calculating the product of the first prediction score sub-vector and the second prediction score sub-vector, and taking the product as the association retrieval score of the to-be-queried action and the to-be-queried location. 3.The instance retrieval method based on cross-semantic attention model according to claim 1, wherein, The step of obtaining the retrieval result corresponding to the to-be-queried action and the to-be-queried location based on the plurality of association retrieval scores comprises: performing a max-pooling operation on the plurality of association retrieval scores to obtain the retrieval result corresponding to the to-be-queried action and the to-be-queried location. 4.The instance retrieval method based on cross-semantic attention model according to claim 1, wherein, Before the step of modifying the location prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the action prediction score vector and the location prediction score vector to obtain the modified location prediction score vector, the method comprises: inputting the action prediction score vector and the location prediction score vector in the training set into a location modification action module to obtain a modified action prediction score vector output by the location modification action module; inputting the action prediction score vector and the location prediction score vector in the training set into an action modification location module to obtain a modified location prediction score vector output by the action modification location module; calculating a first loss between the modified action prediction score vector output by the location modification action module and a preset action score vector; calculating a second loss between the modified location prediction score vector output by the action modification location module and a preset location score vector; substituting the first loss and the second loss into a ninth preset formula to obtain a third loss, wherein the ninth preset formula is as follows: a first loss for representing a difference between the modified action prediction score vector and a preset action score vector in the action modification location module, a second loss for representing a difference between the modified location prediction score vector and a preset location score vector in the action modification location module, and λ for representing a loss fusion weight, the modified action prediction score vector, the modified location prediction score vector, the preset action score vector, the preset location score vector, a third loss; calculating the gradient of the third loss, and updating the parameters of the attention embedding matrix and the parameters of the semantic embedding matrix in the location modification action module and the parameters of the attention embedding matrix and the parameters of the semantic embedding matrix in the action modification location module in a backpropagation gradient manner; returning to execute the step of inputting the action prediction score vector and the location prediction score vector in the training set into the location modification action module to obtain the modified action prediction score vector output by the location modification action module, with a new action prediction score vector in the training set as the action prediction score vector and a new location prediction score vector in the training set as the location prediction score vector; when the number of iterations reaches a preset number, obtaining the trained attention embedding matrix and the trained semantic embedding matrix in the location modification action module and the trained attention embedding matrix and the trained semantic embedding matrix in the action modification location module.

5. A cross-semantic attention model based instance retrieval device employing the cross-semantic attention model based instance retrieval method according to any one of claims 1-4, characterized in that, The instance retrieval device based on the cross-semantic attention model comprises: a prediction score vector acquisition module configured to select any one of the plurality of shots as a target shot, and acquire an action prediction score vector and a location prediction score vector in the target shot; a first modification module configured to modify the location prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the action prediction score vector to obtain a modified location prediction score vector; and a second modification module configured to modify the action prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the location prediction score vector to obtain a modified action prediction score vector. The second correction module is configured to correct the action prediction score vector based on the attention embedding matrix, the semantic embedding matrix, and the location prediction score vector, to obtain a corrected action prediction score vector. The calculation module is configured to calculate an association retrieval score of the query action and the query location based on the corrected action prediction score vector, the corrected location prediction score vector, the query action, and the query location. The loop module is configured to select any shot that has not been selected as a target shot, and return to the step of obtaining the action prediction score vector and the location prediction score vector in the target shot, until all the shots are selected, to obtain a plurality of association retrieval scores. The retrieval result acquisition module is configured to obtain a retrieval result corresponding to the query action and the query location based on the plurality of association retrieval scores. 6.The instance retrieval apparatus based on cross-semantic attention model according to claim 5, wherein, The first correction module is configured to: substitute the action prediction score vector into a first preset formula to calculate an action attention vector and an action semantic vector, wherein the first preset formula is as follows: wherein, for representing an action attention vector, for representing an action prediction score vector, for representing an attention embedding matrix in the action correction location module, , for representing an action semantic vector, for representing a semantic embedding matrix in the action correction location module, , is the total number of action classes, for representing a real-valued matrix of size A by A; substitute the location prediction score vector and the action attention vector into a second preset formula to calculate an action-to-location attention matrix, wherein the second preset formula is as follows: wherein, an attention matrix for representing actions to places, a place prediction score vector; substitute the action semantic vector and the action-to-location attention matrix into a third preset formula to calculate a location prediction score update vector, wherein the third preset formula is as follows: wherein, for representing a location prediction score update vector; substitute the location prediction score update vector and the location prediction score vector into a fourth preset formula to calculate the corrected location prediction score vector, wherein the fourth preset formula is as follows: wherein, for representing the modified location prediction score vector.

7. An instance retrieval device based on a cross-semantic attention model, characterized in that, The instance retrieval device based on the cross-semantic attention model comprises a processor, a memory, and an instance retrieval program based on a cross-semantic attention model stored on the memory and executable by the processor, wherein when the instance retrieval program based on the cross-semantic attention model is executed by the processor, the steps of the instance retrieval method based on the cross-semantic attention model in any one of claims 1 to 4 are implemented.

8. A readable storage medium, characterized by, The readable storage medium stores an instance retrieval program based on a cross-semantic attention model, wherein when the instance retrieval program based on the cross-semantic attention model is executed by the processor, the steps of the instance retrieval method based on the cross-semantic attention model in any one of claims 1 to 4 are implemented.

Images