Cross-modal retrieval method and device, computer device, storage medium and product

Abstract

The application relates to a cross-modal retrieval method and device, computer equipment, a storage medium and a product. The method comprises the following steps: obtaining a text semantic feature and a text global feature corresponding to a target text, and obtaining a plurality of multimedia semantic features and multimedia global features corresponding to a plurality of candidate multimedia data; performing matching processing on the text semantic feature and each multimedia semantic feature respectively to obtain a plurality of target semantic similarities, and performing matching processing on the text global feature and each multimedia global feature respectively to obtain a plurality of target global similarities; determining target multimedia data from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities; and the above method can realize the function of searching a large amount of multimedia data in a power grid digital system, and improves the search efficiency of the large amount of multimedia data in the power grid digital system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a cross-modal retrieval method and device, computer equipment, storage medium and product. BACKGROUND

[0002] In recent years, with the promotion of power grid digitalization construction, the multimedia data (such as video, image, etc.) accumulated in the scenes of unmanned aerial vehicle inspection, online monitoring, robot inspection, etc. has an explosive growth. How to effectively manage the massive multimedia data has become a problem to be solved.

[0003] In some scenarios, sample data is extremely difficult to obtain, and using manual screening or using image retrieval is not a feasible solution. As the most commonly used retrieval method in daily life, text retrieval effectively solves the problem of sample shortage due to its strong description and compatibility.

[0004] Therefore, there is an urgent need for a cross-modal retrieval method to retrieve massive multimedia data in the power grid digitalization system to improve retrieval efficiency. SUMMARY

[0005] Therefore, it is necessary to provide a cross-modal retrieval method, device, computer equipment, storage medium and product capable of improving retrieval efficiency in view of the above technical problems.

[0006] In a first aspect, the present application provides a cross-modal retrieval method. The method comprises:

[0007] Obtaining text semantic features and text global features corresponding to the target text, and obtaining multimedia semantic features and multimedia global features corresponding to a plurality of candidate multimedia data;

[0008] Respectively matching the text semantic features with each multimedia semantic feature to obtain a plurality of target semantic similarities, and respectively matching the text global features with each multimedia global feature to obtain a plurality of target global similarities;

[0009] Based on the plurality of target semantic similarities and the plurality of target global similarities, determining a target multimedia data from the plurality of candidate multimedia data.

[0010] In one embodiment, obtaining the text semantic features corresponding to the target text comprises: extracting a subject field, a predicate field and an object field in the target text to obtain text triple features; and obtaining the text semantic features according to the text triple features.

[0011] In one of the embodiments, the text semantic feature is obtained according to the text triple feature, including: performing global word frequency statistical coding processing on the text triple feature to obtain an initial semantic word vector; performing context feature learning processing on the initial semantic word vector to obtain an intermediate semantic word vector; performing average pooling processing on the intermediate semantic word vector to obtain a target semantic word vector; and performing linear mapping processing on the target semantic word vector to obtain the text semantic feature.

[0012] In one of the embodiments, the process of obtaining the text global feature corresponding to the target text includes: performing global word frequency statistical coding processing on the target text to obtain an initial global word vector; performing context feature learning processing on the initial global word vector to obtain an intermediate global word vector; and performing attention aggregation processing on the intermediate global word vector to obtain the text global feature.

[0013] In one of the embodiments, the multimedia semantic feature corresponding to each candidate multimedia data is obtained according to the initial semantic feature, including: performing pooling processing on the initial semantic feature along the time dimension to obtain an intermediate semantic feature; and performing linear mapping processing on the intermediate semantic feature to obtain the multimedia semantic feature.

[0014] In one of the embodiments, the multimedia semantic feature corresponding to each candidate multimedia data is obtained according to the initial semantic feature, including: performing pooling processing on the initial semantic feature along the time dimension to obtain an intermediate semantic feature; and performing linear mapping processing on the intermediate semantic feature to obtain the multimedia semantic feature.

[0015] In one of the embodiments, the process of obtaining the multimedia global feature corresponding to each candidate multimedia data includes: performing visual feature extraction on the candidate multimedia data by using a visual feature extraction network to obtain initial visual features corresponding to the candidate multimedia data; performing context feature learning processing on the initial visual features to obtain intermediate visual features; and performing attention aggregation processing on the intermediate visual features to obtain the multimedia global feature.

[0016] In one of the embodiments, the text semantic feature includes a plurality of text semantic sub-features, and the multimedia semantic feature includes a plurality of multimedia semantic sub-features; the matching processing is performed on the text semantic feature and each multimedia semantic feature respectively to obtain a plurality of target semantic similarities, including: for each multimedia semantic feature, the cosine similarity between each text semantic sub-feature and each multimedia semantic sub-feature included in the multimedia semantic feature is obtained respectively to obtain a plurality of initial semantic similarities corresponding to each text semantic sub-feature; the intermediate semantic similarity corresponding to each text semantic sub-feature is obtained according to the plurality of initial semantic similarities, and the target semantic similarity between the target text and the multimedia semantic feature is obtained according to the intermediate semantic similarities.

[0017] In one of the embodiments, the intermediate semantic similarity corresponding to each text semantic sub-feature is obtained according to the plurality of initial semantic similarities, including: selecting K initial semantic similarities from the plurality of initial semantic similarities in order of the selection priority of the initial semantic similarities from large to small, and performing weighted average processing on the K initial semantic similarities to obtain the intermediate semantic similarity, the size of the selection priority of the initial semantic similarity being positively correlated with the size of the initial semantic similarity.

[0018] In one of the embodiments, the target semantic similarity between the target text and the multimedia semantic feature is obtained according to the intermediate semantic similarities, including: performing weighted average processing on the intermediate semantic similarities to obtain the target semantic similarity.

[0019] In one of the embodiments, the text global feature is matched with each multimedia global feature to obtain a plurality of target global similarities, including:

[0020] For each multimedia global feature, the cosine similarity between the text global feature and the multimedia global feature is obtained to obtain the target global similarity.

[0021] In one of the embodiments, the target multimedia data is determined from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities, including: for each candidate multimedia data, performing fusion processing on the target semantic similarity and the target global similarity corresponding to the candidate multimedia data to obtain a candidate similarity between the candidate multimedia data and the target text; selecting N candidate similarities from the plurality of candidate similarities in order of the selection priority of the candidate similarities from large to small; and determining N candidate multimedia data corresponding to the N candidate similarities as the target multimedia data.

[0022] In a second aspect, the present application further provides a cross-modal retrieval device. The device comprises:

[0023] The feature acquisition module is configured to acquire text semantic features and text global features corresponding to a target text, and acquire multimedia semantic features and multimedia global features corresponding to a plurality of candidate multimedia data.

[0024] The feature matching module is configured to match the text semantic features with each multimedia semantic feature to obtain a plurality of target semantic similarities, and match the text global feature with each multimedia global feature to obtain a plurality of target global similarities.

[0025] The result determination module is configured to determine target multimedia data from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities.

[0026] In a third aspect, the present application provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0027] obtaining text semantic features and text global features corresponding to the target text, and obtaining multimedia semantic features and multimedia global features corresponding to the plurality of candidate multimedia data;

[0028] performing matching processing on the text semantic features and each of the multimedia semantic features respectively to obtain a plurality of target semantic similarities, and performing matching processing on the text global features and each of the multimedia global features respectively to obtain a plurality of target global similarities;

[0029] determining the target multimedia data from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities.

[0030] In a fourth aspect, the present application provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0031] obtaining text semantic features and text global features corresponding to the target text, and obtaining multimedia semantic features and multimedia global features corresponding to the plurality of candidate multimedia data;

[0032] performing matching processing on the text semantic features and each of the multimedia semantic features respectively to obtain a plurality of target semantic similarities, and performing matching processing on the text global features and each of the multimedia global features respectively to obtain a plurality of target global similarities;

[0033] determining the target multimedia data from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities.

[0034] In a fifth aspect, the present application provides a computer program product. The computer program product comprises a computer program, and the computer program is executed by a processor to implement the following steps:

[0035] obtaining text semantic features and text global features corresponding to the target text, and obtaining multimedia semantic features and multimedia global features corresponding to the plurality of candidate multimedia data;

[0036] performing matching processing on the text semantic features and each of the multimedia semantic features respectively to obtain a plurality of target semantic similarities, and performing matching processing on the text global features and each of the multimedia global features respectively to obtain a plurality of target global similarities;

[0037] The target multimedia data is determined from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities.

[0038] The cross-modal retrieval method, device, computer device, storage medium and product obtain text semantic features and text global features corresponding to the target text, and obtain multimedia semantic features and multimedia global features corresponding to the plurality of candidate multimedia data. The text semantic features are matched with the multimedia semantic features respectively to obtain a plurality of target semantic similarities, and the text global features are matched with the multimedia global features respectively to obtain a plurality of target global similarities. The target multimedia data is determined from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities. Since the target text and the candidate multimedia data are different data modalities and cannot be directly matched, the method obtains the text semantic features and the text global features corresponding to the target text, and obtains the multimedia semantic features and the multimedia global features corresponding to the candidate multimedia data, so as to convert the target text and the candidate multimedia data into features of the same data modality for matching. Meanwhile, the method considers the different feature granularities, matches the text global features and the multimedia global features of coarse granularity, and matches the text semantic features and the multimedia semantic features of fine granularity to obtain the plurality of target semantic similarities and the plurality of target global similarities respectively. The plurality of target semantic similarities can accurately measure the similarity between the target text and the plurality of candidate multimedia data at the fine granularity, and the plurality of target global features can accurately measure the similarity between the target text and the plurality of candidate multimedia data at the coarse granularity. Then, the target multimedia data is determined from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities, the function of retrieving the massive multimedia data in the power grid digital system is realized, and the retrieval efficiency of the massive multimedia data in the power grid digital system is improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 An application environment diagram of the cross-modal retrieval method in an embodiment;

[0040] Figure 2 A flowchart of the cross-modal retrieval method in an embodiment;

[0041] Figure 3 A flowchart of the step of obtaining the text semantic features corresponding to the target text in an embodiment;

[0042] Figure 4 A flowchart of the step of obtaining the text semantic features according to the warm-mix ternary feature in an embodiment;

[0043] Figure 5A flowchart of a process of acquiring the text global feature corresponding to the target text in an embodiment;

[0044] Figure 6 A flowchart of a process of acquiring the multimedia semantic features corresponding to the plurality of candidate multimedia data in an embodiment;

[0045] Figure 7 A flowchart of a process of acquiring the multimedia semantic features corresponding to the candidate multimedia data according to the initial semantic features in an embodiment;

[0046] Figure 8 A flowchart of a process of acquiring the multimedia global features corresponding to the plurality of candidate multimedia data in an embodiment;

[0047] Figure 9 A flowchart of a process of determining the target semantic similarity in an embodiment;

[0048] Figure 10 A flowchart of a process of determining the target multimedia data in an embodiment;

[0049] Figure 11 A flowchart of a process of a cross-modal retrieval method in another embodiment;

[0050] Figure 12 A block diagram of a structure of a cross-modal retrieval device in an embodiment;

[0051] Figure 13 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0052] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0053] The cross-modal retrieval method provided by the embodiments of the present application can be applied to, for example, Figure 1The application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart TV, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented by a stand-alone server or a server cluster composed of multiple servers.

[0054] The terminal 102 obtains the text semantic features and the text global features corresponding to the target text, and obtains the multimedia semantic features and the multimedia global features corresponding to the plurality of candidate multimedia data; the text semantic features are matched with each multimedia semantic feature respectively to obtain a plurality of target semantic similarities, and the text global features are matched with each multimedia global feature respectively to obtain a plurality of target global similarities; and the target multimedia data is determined from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities.

[0055] In one embodiment, as shown in Figure 2 , a cross-modal retrieval method is provided. The method is applied to the terminal 102 in Figure 1 for example, which includes the following steps:

[0056] Step 202, obtaining the text semantic features and the text global features corresponding to the target text, and obtaining the multimedia semantic features and the multimedia global features corresponding to the plurality of candidate multimedia data.

[0057] Among them, the target text refers to the information to be retrieved input to the terminal through the external input device of the terminal; for example, the target text can be text form information to be retrieved converted from voice retrieval information input through the voice input device of the terminal; the target text can also be text form information to be retrieved input directly through the text input device of the terminal. The candidate multimedia data refers to each image data or video data to be matched in the target multimedia database; in the embodiment of the application, the terminal determines the target multimedia data, i.e. the retrieval result, from the plurality of candidate multimedia data according to the obtained target text. Among them, the target multimedia database can be a multimedia database in the power grid digital system, and the corresponding target text can be information to be retrieved in the power grid scene.

[0058] The text semantic feature refers to a feature corresponding to a key field in the target text; the target global feature refers to a feature corresponding to the whole target text; the multimedia semantic feature refers to a feature corresponding to key semantic information extracted from the candidate multimedia data; and the multimedia global feature refers to a feature corresponding to the whole candidate multimedia data.

[0059] For example, the text semantic feature can be obtained by performing dependency syntax analysis on the target text; the text global feature can be obtained by performing global feature extraction on the target text; the multimedia semantic feature can be obtained by performing scene graph analysis on the candidate multimedia data; and the multimedia global feature can be obtained by performing global feature extraction on the multimedia data.

[0060] Text and multimedia data are two different data modalities and cannot be directly matched. Therefore, in the embodiments of the present application, the text semantic feature and the text global feature corresponding to the target text are obtained, and the multimedia semantic feature and the multimedia global feature corresponding to the candidate multimedia data are obtained, and the features with the same data modality are used for matching to match the target text and each candidate multimedia data, so as to determine the retrieval result corresponding to the target text, i.e., the target multimedia data.

[0061] In a possible implementation, the multimedia semantic feature and the multimedia global feature corresponding to each candidate multimedia data can be pre-extracted according to a preset method and stored in a target database; when the terminal obtains the target text to be retrieved, the stored multimedia semantic features and multimedia global features are directly read for matching. In another possible implementation, when the terminal obtains the target text to be retrieved, the corresponding text semantic feature and text global feature are extracted from the target text, and the corresponding multimedia semantic feature and multimedia global feature are extracted from each candidate multimedia data.

[0062] In step 204, the text semantic feature is matched with each multimedia semantic feature to obtain a plurality of target semantic similarities, and the text global feature is matched with each multimedia global feature to obtain a plurality of target global similarities.

[0063] The text semantic feature and the multimedia semantic feature have similar structures at the feature level, and the text global feature and the multimedia global feature have similar structures at the feature level. The feature granularity of the text semantic feature corresponding to the target text is smaller than that of the text global feature, and similarly, the feature granularity of the multimedia semantic feature corresponding to the candidate multimedia data is smaller than that of the multimedia global feature. In the embodiments of the present application, the text semantic feature and the multimedia semantic feature are referred to as a fine-grained feature, and the text global feature and the multimedia global feature are referred to as a coarse-grained feature.

[0064] The embodiment of the present application considers different feature granularities, and performs matching processing on coarse-grained text global features and multimedia global features, and performs matching processing on fine-grained text semantic features and multimedia semantic features, to obtain a plurality of target semantic similarities and a plurality of target global similarities, so that the plurality of target semantic similarities can accurately measure the similarity between the target text and the plurality of candidate multimedia data in fine granularity, and so that the plurality of target global similarities can accurately measure the similarity between the target text and the plurality of candidate multimedia data in coarse granularity.

[0065] In step 206, the target multimedia data is determined from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities.

[0066] The number of target multimedia data is determined according to the search requirement. For example, the number of target multimedia data can be one or multiple.

[0067] For example, if the current search requirement is to find multimedia data that is most consistent with the content of the target text from the target database, the candidate multimedia data with the highest similarity to the target text is determined as the target multimedia data from the plurality of candidate multimedia data according to the plurality of target semantic similarities and the plurality of target global similarities.

[0068] For example, if the current search requirement is to find N multimedia data that is consistent with the content of the target text from the target database, the candidate multimedia data in the front N positions can be determined as the target multimedia data from the plurality of candidate multimedia data according to the plurality of target semantic similarities and the plurality of target global similarities, and then sorted in descending order of similarity. N is a positive integer greater than 1.

[0069] In the cross-modal retrieval method, the text semantic feature and the text global feature corresponding to the target text are obtained, and the multimedia semantic feature and the multimedia global feature corresponding to the plurality of candidate multimedia data are obtained; the text semantic feature is matched with each multimedia semantic feature respectively to obtain a plurality of target semantic similarities, and the text global feature is matched with each multimedia global feature respectively to obtain a plurality of target global similarities; and the target multimedia data is determined from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities. Since the target text and each candidate multimedia data are different data modalities and cannot be directly matched, the above method obtains the text semantic feature and the text global feature corresponding to the target text, and obtains the multimedia semantic feature and the multimedia global feature corresponding to each candidate multimedia data, so as to convert the target text and each candidate multimedia data into features with the same data modality for matching. Meanwhile, the method considers the difference in feature granularity, matches the text global feature and the multimedia global feature with coarse granularity, and matches the text semantic feature and the multimedia semantic feature with fine granularity, to obtain the plurality of target semantic similarities and the plurality of target global similarities respectively, so that the plurality of target semantic similarities can accurately measure the similarity between the target text and the plurality of candidate multimedia data at the fine granularity, and the plurality of target global features can accurately measure the similarity between the target text and the plurality of candidate multimedia data at the coarse granularity, and then the target multimedia data is determined from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities, to realize the function of retrieving the massive multimedia data in the power grid digital system, and improve the retrieval efficiency of the massive multimedia data in the power grid digital system.

[0070] In one embodiment, based on Figure 2 as shown in the embodiment, please refer to Figure 3 The embodiment relates to how to obtain the text semantic feature corresponding to the target text. As shown in Figure 3 The embodiment provides a process for obtaining the text semantic feature corresponding to the target text, which includes the following steps:

[0071] In step 302, the subject field, the predicate field and the object field in the target text are extracted to obtain the text triple feature.

[0072] The triple refers to a data structure in the form of <X, Y, Z>. The text triple feature in the embodiment refers to a feature vector in the form of <subject field, predicate field, object field> extracted from the target text. The features representing the subject, the predicate and the object can also be extracted from the multimedia data. Therefore, the triple feature in the form of <subject, predicate, object> is used as the feature structure of the text semantic feature and the multimedia semantic feature in the embodiment.

[0073] For example, dependency parsing can be used to perform syntactic analysis on the target text to obtain the part-of-speech tag for each word. Then, for each verb in the target text, nuubj is selected to find the subject corresponding to the verb, and obj is selected to find the object corresponding to the verb, thus obtaining the text triplet features corresponding to each verb. The number of text triplet features corresponding to the target text is greater than or equal to 1.

[0074] Step 304: Obtain text semantic features based on text triple features.

[0075] Among them, the text triple features are specific text fields that cannot be directly recognized by the terminal or used to match the features extracted from the candidate multimedia data. Therefore, it is necessary to convert the text triple features into a form that the terminal can directly recognize and use to match the features extracted from the candidate multimedia data.

[0076] In one possible implementation method, please refer to Figure 4 Based on the text triple features, semantic features of the text are obtained, including:

[0077] Step 402: Perform global word frequency statistical encoding on the text triplet features to obtain the initial semantic word vector.

[0078] For example, for each word in the text triplet features, the GloVe (Global Vectors for Word Representation) method is used for encoding to obtain the initial semantic word vector {w1, w2, ..., w} corresponding to the text triplet features. m} where m is a natural number greater than 1. GloVe can represent a word as a vector of real numbers.

[0079] Step 404: Perform context feature learning processing on the initial semantic word vectors to obtain intermediate semantic word vectors.

[0080] For example, a bidirectional GRU (Gate Recurrent Unit) can be used to process the initial semantic word vectors to obtain intermediate semantic word vectors that contain contextual information.

[0081] Step 406: Perform average pooling on the intermediate semantic word vectors to obtain the target semantic word vectors.

[0082] For example, an average pooling network is used to perform average pooling on the intermediate semantic word vectors to obtain the target semantic word vector {s}. s ,s v ,s o}, where ss corresponding subject field, s v corresponding predicate field, s o corresponding object field.

[0083] At step 408, linear mapping processing is performed on the target semantic word vector to obtain a text semantic feature.

[0084] Exemplarily, the target semantic word vector is processed into a text semantic feature using linear mapping, as shown in the following formula:

[0085]

[0086] wherein s f represents a text semantic feature, is a trainable weight, b s is a corresponding bias.

[0087] For a target text, the text semantic feature can be represented as wherein Q is the number of text semantic features corresponding to the target text, and Q is an integer greater than or equal to 1. When the number of text semantic features corresponding to the target text is greater than 1, in order to facilitate distinction, in some embodiments, each text semantic feature can also be referred to as a text semantic sub-feature, and the overall semantic feature corresponding to the target text can be referred to as a text semantic feature.

[0088] In this embodiment, the text triple feature is obtained by extracting the subject field, predicate field and object field in the target text, and the text semantic feature of the target text is obtained based on the text triple feature in the form of <subject field, predicate field, object field>. The text semantic feature can represent the main syntactic component of the target text and better represent the semantic level features of the target text.

[0089] In one embodiment, based on Figure 2 as shown in the embodiment, please refer to Figure 5 The present embodiment relates to the process of obtaining the text global feature corresponding to the target text. As Figure 5 shown, the process of obtaining the text global feature corresponding to the target text provided by the present embodiment includes:

[0090] At step 502, global word frequency statistical encoding processing is performed on the target text to obtain an initial global word vector.

[0091] Exemplarily, for each word in the target text, GloVe method is used for encoding to obtain an initial global word vector corresponding to the target text.

[0092] At step 504, context feature learning processing is performed on the initial global word vector to obtain an intermediate global word vector.

[0093] Exemplarily, the initial global word vector can be processed using a bidirectional GRU to obtain an intermediate global word vector containing context information.

[0094] In step 506, the intermediate global word vector is subjected to attention aggregation processing to obtain a text global feature.

[0095] Exemplarily, the intermediate global word vector is aggregated using an attention mechanism to obtain the text global feature. The number of text global features corresponding to the target text is 1.

[0096] In one embodiment, based on Figure 2 As shown in the embodiment, please refer to Figure 6 The embodiment relates to a process for obtaining multimedia semantic features corresponding to a plurality of candidate multimedia data. As shown in the embodiment, Figure 6 The process for obtaining multimedia semantic features corresponding to a plurality of candidate multimedia data provided by the embodiment includes the following steps:

[0097] In step 602, for each candidate multimedia data, subject features, predicate features and object features of the candidate multimedia data are extracted to obtain initial semantic features.

[0098] In a possible implementation method, the candidate multimedia data is image data, a scene graph parser is used to parse visual relationships in the candidate multimedia data, subject features, predicate features and object features of the candidate multimedia data are extracted according to the visual relationships, and the initial semantic features of the candidate multimedia data are obtained in the form of a triple of <subject features, predicate features, object features>.

[0099] In a possible implementation method, the candidate multimedia data is video data, a plurality of video sampling frames of the candidate multimedia data are obtained first, a scene graph parser is used to parse each of the plurality of video sampling frames to obtain visual relationships corresponding to each video sampling frame; the same objects between the plurality of video sampling frames are merged to obtain an object sequence; and the plurality of visual relationships are associated with the object sequence to obtain the initial semantic features corresponding to the candidate multimedia data.

[0100] The number of initial semantic features can be multiple, and each initial semantic feature is in the form of a triple of <subject features, predicate features, object features>, wherein the subject features and the object features are represented by the bounding boxes of the corresponding objects, and the predicate features are represented by the union bounding box of the corresponding subject features and object features.

[0101] In step 604, the multimedia semantic features corresponding to the candidate multimedia data are obtained according to the initial semantic features.

[0102] In a possible implementation method, the candidate multimedia data is video data, please refer toFigure 7 According to the initial semantic feature, a multimedia semantic feature corresponding to the candidate multimedia data is obtained, including:

[0103] In step 702, the initial semantic feature is pooled along the time dimension to obtain an intermediate semantic feature.

[0104] For example, when the candidate multimedia data is video data, the initial semantic feature corresponding to the candidate multimedia data is associated with an object sequence. The initial semantic feature is pooled along the time dimension to obtain an intermediate semantic feature {v s ,v v ,v o}, where v s corresponds to a subject feature, v v corresponds to a predicate feature, and v o corresponds to an object feature.

[0105] In step 704, the intermediate semantic feature is linearly mapped to obtain a multimedia semantic feature.

[0106] For example, the intermediate semantic feature is linearly mapped to the multimedia semantic feature, as shown in the following formula:

[0107]

[0108] where v f represents a multimedia semantic feature, is a trainable weight, and b v is a corresponding bias. For video data form candidate multimedia data, the multimedia semantic feature can be represented as where P is the number of multimedia semantic features corresponding to the candidate multimedia data, and P is an integer greater than or equal to 1. When the number of multimedia semantic features corresponding to the candidate multimedia data is greater than 1, in order to facilitate the distinction, in some embodiments, each multimedia semantic feature can also be referred to as a multimedia semantic sub-feature, and the overall semantic feature corresponding to the candidate multimedia data can be referred to as a multimedia semantic feature.

[0109] In one possible implementation method, the candidate multimedia data is image data. At this time, the initial semantic feature is pooled to obtain an intermediate semantic feature, and then the intermediate semantic feature is linearly mapped to obtain the multimedia semantic feature corresponding to the candidate multimedia data.

[0110] In this embodiment, the subject feature, the predicate feature and the object feature of each candidate multimedia data are extracted to obtain initial semantic features, and then the multimedia semantic features corresponding to the candidate multimedia data are obtained according to the initial semantic features. In this way, the subject, the predicate and the object features of the candidate multimedia data can better represent the semantic features of the candidate multimedia data, and the semantic features of the text retrieval can be better matched, thereby improving the retrieval accuracy.

[0111] In one embodiment, based on Figure 2 As shown in the embodiment shown in Figure 8 The embodiment relates to a process for obtaining multimedia global features corresponding to a plurality of candidate multimedia data. As shown in Figure 8 The process for obtaining multimedia global features corresponding to a plurality of candidate multimedia data provided by the embodiment includes the following steps.

[0112] In step 802, the visual feature extraction network is used to extract visual features of the candidate multimedia data to obtain initial visual features corresponding to the candidate multimedia data.

[0113] For example, the candidate multimedia data is in the form of video data, and the frame-level visual feature extraction network ResNet-152 is used to extract visual features of the candidate multimedia data to obtain initial visual features corresponding to the candidate multimedia data, and the initial visual features are frame-level initial visual features.

[0114] For example, the candidate multimedia data is in the form of image data, and the visual feature extraction network is used to extract visual features of the candidate multimedia data to obtain initial visual features corresponding to the candidate multimedia data.

[0115] In step 804, the initial visual features are subjected to context feature learning processing to obtain intermediate visual features.

[0116] For example, the initial visual features are subjected to context feature learning processing by using the bidirectional GRU network to obtain intermediate visual features containing context information.

[0117] In step 806, the intermediate visual features are subjected to attention aggregation processing to obtain multimedia global features.

[0118] For example, the attention mechanism is used to aggregate the intermediate visual features to obtain multimedia global features corresponding to the candidate multimedia data.

[0119] In one embodiment, based on Figure 2 As shown in the embodiment shown in Figure 9This embodiment involves matching text semantic features with various multimedia semantic features to obtain multiple target semantic similarities. In this embodiment, text semantic features include multiple text semantic sub-features; multimedia semantic features include multiple multimedia semantic sub-features, such as... Figure 9 As shown, the process of matching the text semantic features provided in this embodiment with the semantic features of various multimedia to obtain multiple target semantic similarities includes:

[0120] Step 902: For each multimedia semantic feature, obtain the cosine similarity between each text semantic sub-feature and each multimedia semantic sub-feature included in the multimedia semantic feature, and obtain multiple initial semantic similarities corresponding to each text semantic sub-feature.

[0121] Step 904: Based on multiple initial semantic similarities, obtain the intermediate semantic similarity corresponding to each text semantic sub-feature, and obtain the target semantic similarity between the target text and the multimedia semantic features based on each intermediate semantic similarity.

[0122] In one possible implementation, the intermediate semantic similarity corresponding to each text semantic sub-feature is obtained based on multiple initial semantic similarities, including: selecting K initial semantic similarities from multiple initial semantic similarities in descending order of their selection priority, and performing a weighted average of the K initial semantic similarities to obtain the intermediate semantic similarity. The selection priority of the initial semantic similarities is positively correlated with the size of the initial semantic similarities.

[0123] For example, the process of obtaining intermediate semantic similarity can be represented by the following formula:

[0124]

[0125] in, Representing text semantic sub-features The intermediate semantic similarity with the multimedia semantic feature v is calculated by AVG (average) and TopK (top K) (taking the K largest values ​​in the sequence). Representing multimedia semantic sub-features Textual semantic sub-features The cosine similarity, where P is the number of multimedia semantic sub-features.

[0126] In one possible implementation, the target semantic similarity between the target text and the multimedia semantic features is obtained based on each intermediate semantic similarity, including: performing a weighted average of each intermediate semantic similarity to obtain the target semantic similarity.

[0127] For example, the average value of the intermediate semantic similarities of all text semantic sub-features is taken as the target semantic similarity, which can be expressed by the following formula:

[0128]

[0129] wherein S f represents the target semantic similarity, and Q represents the number of text semantic sub-features.

[0130] In one embodiment, based on Figure 2 the embodiment shown in FIG. 6, the embodiment relates to a process of matching the text global feature with each multimedia global feature to obtain a plurality of target global similarities. In the embodiment, the process of matching the text global feature with each multimedia global feature to obtain a plurality of target global similarities includes: for each multimedia global feature, obtaining the cosine similarity between the text global feature and the multimedia global feature to obtain a target global similarity; for example, the process can be expressed by the following formula:

[0131] S c = cos(v c , s c ),

[0132] wherein S c represents the target global similarity, and cos(v c , s c ) represents the cosine similarity of the text global feature s c and the multimedia global feature v c .

[0133] In one embodiment, based on Figure 2 the embodiment shown in FIG. 6, please refer to Figure 10 , the embodiment relates to a process of determining the target multimedia data from a plurality of candidate multimedia data based on a plurality of target semantic similarities and a plurality of target global similarities. As Figure 10 shown in FIG. 7, the embodiment relates to a process of determining the target multimedia data from a plurality of candidate multimedia data based on a plurality of target semantic similarities and a plurality of target global similarities, which includes:

[0134] Step 1002, for each candidate multimedia data, the target semantic similarity and the target global similarity corresponding to the candidate multimedia data are fused to obtain a candidate similarity between the candidate multimedia data and the target text.

[0135] For example, for each candidate multimedia data, the average value of the target semantic similarity and the target global similarity is determined as the candidate similarity between the candidate multimedia data and the target text; which can be expressed by the following formula:

[0136] S(v,s) = (S c + S f ) / 2,

[0137] wherein S(v,s) represents a candidate similarity of the target text s and the v of the candidate multimedia data.

[0138] Step 1004, according to the screening priority of each candidate similarity from large to small, N candidate similarities are screened from the plurality of candidate similarities in turn.

[0139] Step 1004, according to the screening priority of each candidate similarity from large to small, N candidate similarities are screened from the plurality of candidate similarities in turn.

[0140] Step 1006, the N candidate multimedia data corresponding to the N candidate similarities are determined as the target multimedia data.

[0141] For example, when N is 1, the candidate similarity corresponding to the maximum value is found from each candidate similarity, and the candidate multimedia data corresponding to the candidate similarity is determined as the target multimedia data. For example, when N is 5, the 5 candidate similarities with the largest values are found from each candidate similarity, and the candidate multimedia data corresponding to the 5 candidate similarities is determined as the target multimedia data.

[0142] In one embodiment, as shown in Figure 11 , a modal retrieval method is provided, which is applied to the terminal 102 in Figure 1 as an example, including the following steps:

[0143] Step 1102, obtaining a target text.

[0144] Step 1104, extracting the subject field, predicate field and object field in the target text to obtain a text triple feature.

[0145] Step 1106, obtaining a text semantic feature according to the text triple feature.

[0146] Optionally, the text triple feature is subjected to global word frequency statistical coding processing to obtain an initial semantic word vector; the initial semantic word vector is subjected to context feature learning processing to obtain an intermediate semantic word vector; the intermediate semantic word vector is subjected to average pooling processing to obtain a target semantic word vector; and the target semantic word vector is subjected to linear mapping processing to obtain the text semantic feature.

[0147] Step 1108, obtaining a text global feature corresponding to the target text.

[0148] Optionally, the text global feature corresponding to the target text is obtained, including: performing global word frequency statistical coding processing on the target text to obtain an initial global word vector; performing context feature learning processing on the initial global word vector to obtain an intermediate global word vector; and performing attention aggregation processing on the intermediate global word vector to obtain the text global feature.

[0149] In step 1110, the multimedia semantic features and the multimedia global features corresponding to the plurality of candidate multimedia data are obtained.

[0150] Optionally, the multimedia semantic features corresponding to the plurality of candidate multimedia data are obtained, including: for each candidate multimedia data, extracting subject features, predicate features and object features of the candidate multimedia data to obtain initial semantic features; and obtaining the multimedia semantic features corresponding to the candidate multimedia data according to the initial semantic features.

[0151] Optionally, the multimedia semantic features corresponding to the candidate multimedia data are obtained according to the initial semantic features, including: performing pooling processing on the initial semantic features along a time dimension to obtain intermediate semantic features; and performing linear mapping processing on the intermediate semantic features to obtain the multimedia semantic features.

[0152] Optionally, the process of obtaining the multimedia global features corresponding to the plurality of candidate multimedia data includes: for each candidate multimedia data, performing visual feature extraction on the candidate multimedia data by using a visual feature extraction network to obtain initial visual features corresponding to the candidate multimedia data; performing context feature learning processing on the initial visual features to obtain intermediate visual features; and performing attention aggregation processing on the intermediate visual features to obtain the multimedia global features.

[0153] In step 1112, the text semantic features are matched with each of the multimedia semantic features respectively to obtain a plurality of target semantic similarities.

[0154] Optionally, the text semantic features include a plurality of text semantic sub-features, and the multimedia semantic features include a plurality of multimedia semantic sub-features.

[0155] The text semantic features are matched with each of the multimedia semantic features respectively to obtain a plurality of target semantic similarities, including: for each multimedia semantic feature, respectively obtaining cosine similarities between each text semantic sub-feature and each multimedia semantic sub-feature included in the multimedia semantic feature to obtain a plurality of initial semantic similarities corresponding to each text semantic sub-feature; obtaining an intermediate semantic similarity corresponding to each text semantic sub-feature according to the plurality of initial semantic similarities, and obtaining a target semantic similarity between the target text and the multimedia semantic feature according to the intermediate semantic similarities.

[0156] Optionally, the intermediate semantic similarity corresponding to each text semantic sub-feature is obtained according to the plurality of initial semantic similarities, including: K initial semantic similarities are selected from the plurality of initial semantic similarities in order of screening priority of each initial semantic similarity from large to small, and the K initial semantic similarities are weighted and averaged to obtain the intermediate semantic similarity, and the size of the screening priority of the initial semantic similarity is positively correlated with the size of the initial semantic similarity.

[0157] Optionally, the target semantic similarity of the target text and the multimedia semantic feature is obtained according to each intermediate semantic similarity, including: the intermediate semantic similarities are weighted and averaged to obtain the target semantic similarity.

[0158] Step 1114, the text global feature is matched with each multimedia global feature respectively, and a plurality of target global similarities are obtained.

[0159] Optionally, the text global feature is matched with each multimedia global feature respectively, and a plurality of target global similarities are obtained, including: for each multimedia global feature, the cosine similarity between the text global feature and the multimedia global feature is obtained to obtain the target global similarity.

[0160] Step 1116, based on the plurality of target semantic similarities and the plurality of target global similarities, the target multimedia data is determined from the plurality of candidate multimedia data.

[0161] Optionally, the target multimedia data is determined from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities, including: for each candidate multimedia data, the target semantic similarity and the target global similarity corresponding to the candidate multimedia data are fused to obtain a candidate similarity between the candidate multimedia data and the target text; K candidate similarities are selected from the plurality of candidate similarities in order of screening priority of each candidate similarity from large to small; and N candidate multimedia data corresponding to the N candidate similarities are determined as the target multimedia data.

[0162] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0163] Based on the same inventive concept, the embodiments of the present application also provide a cross-modal retrieval device for implementing the above-mentioned cross-modal retrieval method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more cross-modal retrieval device embodiments provided below can refer to the limitations of the cross-modal retrieval method in the above text, which will not be repeated here.

[0164] In one embodiment, as shown in Figure 12 a cross-modal retrieval device is provided, comprising: a feature acquisition module 1202, a feature matching module 1204, and a result determination module 1206, wherein:

[0165] The feature acquisition module 1202 is configured to acquire text semantic features and text global features corresponding to the target text, and acquire multimedia semantic features and multimedia global features corresponding to a plurality of candidate multimedia data.

[0166] The feature matching module 1204 is configured to perform matching processing on the text semantic features and each multimedia semantic feature respectively to obtain a plurality of target semantic similarities, and perform matching processing on the text global features and each multimedia global feature respectively to obtain a plurality of target global similarities.

[0167] The result determination module 1206 is configured to determine the target multimedia data from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities.

[0168] In one embodiment, the feature acquisition module 1202 is configured to extract a subject field, a predicate field and an object field in the target text to obtain text triple features, and to acquire the text semantic features according to the text triple features.

[0169] In an embodiment, the feature obtaining module 1202 is configured to perform global word frequency statistical encoding processing on the text triple feature to obtain an initial semantic word vector, perform context feature learning processing on the initial semantic word vector to obtain an intermediate semantic word vector, perform average pooling processing on the intermediate semantic word vector to obtain a target semantic word vector, and perform linear mapping processing on the target semantic word vector to obtain the text semantic feature.

[0170] In an embodiment, the feature obtaining module 1202 is configured to perform global word frequency statistical encoding processing on the target text to obtain an initial global word vector, perform context feature learning processing on the initial global word vector to obtain an intermediate global word vector, and perform attention aggregation processing on the intermediate global word vector to obtain the text global feature.

[0171] In an embodiment, the feature obtaining module 1202 is configured to, for each candidate multimedia data, extract subject feature, predicate feature and object feature of the candidate multimedia data to obtain initial semantic features, and obtain the multimedia semantic feature corresponding to the candidate multimedia data according to the initial semantic features.

[0172] In an embodiment, the feature obtaining module 1202 is configured to perform pooling processing on the initial semantic features along the time dimension to obtain intermediate semantic features, and perform linear mapping processing on the intermediate semantic features to obtain the multimedia semantic feature.

[0173] In an embodiment, the feature obtaining module 1202 is configured to, for each candidate multimedia data, perform visual feature extraction on the candidate multimedia data by using a visual feature extraction network to obtain initial visual features corresponding to the candidate multimedia data, perform context feature learning processing on the initial visual features to obtain intermediate visual features, and perform attention aggregation processing on the intermediate visual features to obtain the multimedia global feature.

[0174] In an embodiment, the text semantic feature includes a plurality of text semantic sub-features, and the multimedia semantic feature includes a plurality of multimedia semantic sub-features; the feature matching module 1204 is configured to, for each multimedia semantic feature, obtain a plurality of initial semantic similarities corresponding to each text semantic sub-feature, respectively, between each text semantic sub-feature and each multimedia semantic sub-feature included in the multimedia semantic feature, obtain an intermediate semantic similarity corresponding to each text semantic sub-feature according to the plurality of initial semantic similarities, and obtain a target semantic similarity between the target text and the multimedia semantic feature according to the intermediate semantic similarities.

[0175] In an embodiment, the feature matching module 1204 is configured to sequentially select K initial semantic similarities from the plurality of initial semantic similarities in a descending order of selection priorities of the initial semantic similarities, and perform weighted average processing on the K initial semantic similarities to obtain an intermediate semantic similarity, wherein the selection priorities of the initial semantic similarities are positively correlated with the sizes of the initial semantic similarities.

[0176] In an embodiment, the feature matching module 1204 is configured to perform weighted average processing on the intermediate semantic similarities to obtain a target semantic similarity.

[0177] In an embodiment, the feature matching module 1204 is configured to obtain, for each multimedia global feature, a cosine similarity between the text global feature and the multimedia global feature to obtain a target global similarity.

[0178] In an embodiment, the result determining module 1206 is configured to, for each candidate multimedia data, perform fusion processing on the target semantic similarity and the target global similarity corresponding to the candidate multimedia data to obtain a candidate similarity between the candidate multimedia data and the target text; sequentially select N candidate similarities from the plurality of candidate similarities in a descending order of selection priorities of the candidate similarities; and determine N candidate multimedia data corresponding to the N candidate similarities as the target multimedia data.

[0179] The above modules in the cross-modal retrieval apparatus can be realized by software, hardware, or a combination thereof. The above modules can be embedded in or independent of a processor in a computer device in a hardware form, or stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to the above modules.

[0180] In an embodiment, a computer device is provided, which can be a server. An internal structure diagram of the computer device can be as shown in FIG. 12. Figure 13As shown in the figure. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store candidate multimedia data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a cross-modal retrieval method.

[0181] Those skilled in the art can understand that, Figure 13 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0182] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the above method embodiments.

[0183] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0184] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0185] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0186] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of semantic database and non-semantic database. The non-semantic database can include a distributed database based on a block chain, etc., without limitation. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without limitation.

[0187] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0188] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A cross-modal retrieval method, characterized in that, The method comprises: obtaining text semantic features and text global features corresponding to a target text, and obtaining multimedia semantic features and multimedia global features corresponding to a plurality of candidate multimedia data, wherein, for each of the candidate multimedia data, subject features, predicate features and object features of the candidate multimedia data are extracted to obtain initial semantic features, the initial semantic features are pooled along a time dimension to obtain intermediate semantic features, and the intermediate semantic features are linearly mapped to obtain the multimedia semantic features; matching the text semantic features with each of the multimedia semantic features respectively to obtain a plurality of target semantic similarities, and matching the text global features with each of the multimedia global features respectively to obtain a plurality of target global similarities; determining a target multimedia data from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities.

2. The method of claim 1, wherein, The method comprises: extracting subject fields, predicate fields and object fields in the target text to obtain text triple features; obtaining the text semantic features according to the text triple features.

3. The method of claim 2, wherein, The method comprises: globally word frequency statistical encoding the text triple features to obtain initial semantic word vectors; context feature learning the initial semantic word vectors to obtain intermediate semantic word vectors; average pooling the intermediate semantic word vectors to obtain target semantic word vectors; linearly mapping the target semantic word vectors to obtain the text semantic features.

4. The method of claim 1, wherein, The method comprises: globally word frequency statistical encoding the target text to obtain initial global word vectors; context feature learning the initial global word vectors to obtain intermediate global word vectors; attention aggregation the intermediate global word vectors to obtain the text global features.

5. The method of claim 1, wherein, The method comprises: for each of the candidate multimedia data, performing visual feature extraction on the candidate multimedia data by using a visual feature extraction network to obtain initial visual features corresponding to the candidate multimedia data; context feature learning the initial visual features to obtain intermediate visual features; attention aggregation the intermediate visual features to obtain the multimedia global features.

6. The method of claim 1, wherein, The text semantic features comprise a plurality of text semantic sub-features, and the multimedia semantic features comprise a plurality of multimedia semantic sub-features; the matching the text semantic features with each of the multimedia semantic features respectively to obtain a plurality of target semantic similarities comprises: for each of the multimedia semantic features, obtaining a plurality of initial semantic similarities corresponding to each of the text semantic sub-features between each of the text semantic sub-features and each of the multimedia semantic sub-features included in the multimedia semantic features; According to the plurality of initial semantic similarities, an intermediate semantic similarity corresponding to each of the text semantic sub-features is obtained, and the target semantic similarity of the target text and the multimedia semantic feature is obtained according to the intermediate semantic similarities.

7. The method of claim 6, wherein, The obtaining of the intermediate semantic similarity corresponding to each of the text semantic sub-features according to the plurality of initial semantic similarities comprises: K initial semantic similarities are sequentially selected from the plurality of initial semantic similarities in a descending order of selection priorities of the initial semantic similarities, and the K initial semantic similarities are weighted and averaged to obtain the intermediate semantic similarity, wherein the selection priority of the initial semantic similarity is positively correlated with the size of the initial semantic similarity.

8. The method according to claim 6 or 7, characterized in that, The obtaining of the target semantic similarity of the target text and the multimedia semantic feature according to the intermediate semantic similarities comprises: The target semantic similarity is obtained by weighted and averaging the intermediate semantic similarities.

9. The method of claim 1, wherein, The matching of the text global feature with each of the multimedia global features to obtain a plurality of target global similarities comprises: For each of the multimedia global features, a cosine similarity between the text global feature and the multimedia global feature is obtained to obtain the target global similarity.

10. The method of claim 1, wherein, The determination of the target multimedia data from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities comprises: For each of the candidate multimedia data, a target semantic similarity and a target global similarity corresponding to the candidate multimedia data are fused to obtain a candidate similarity between the candidate multimedia data and the target text; N candidate similarities are sequentially selected from the plurality of candidate similarities in a descending order of selection priority sizes of the candidate similarities; N candidate multimedia data corresponding to the N candidate similarities are determined as the target multimedia data.

11. A cross-modal retrieval apparatus, comprising: The apparatus comprises: A feature obtaining module is configured to obtain a text semantic feature and a text global feature corresponding to a target text, and obtain a multimedia semantic feature and a multimedia global feature corresponding to a plurality of candidate multimedia data, wherein, for each of the candidate multimedia data, a subject feature, a predicate feature and an object feature of the candidate multimedia data are extracted to obtain an initial semantic feature, the initial semantic feature is subjected to a pooling process along a time dimension to obtain an intermediate semantic feature, and the intermediate semantic feature is subjected to a linear mapping process to obtain the multimedia semantic feature; A feature matching module is configured to match the text semantic feature with each of the multimedia semantic features to obtain a plurality of target semantic similarities, and match the text global feature with each of the multimedia global features to obtain a plurality of target global similarities; A result determining module is configured to determine a target multimedia data from the plurality of candidate multimedia data based on the plurality of target semantic similarities and the plurality of target global similarities.

12. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 10.

13. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 10.

Images