Query method, model training method, device, equipment and storage medium

Abstract

The embodiments of the present application provide a query method, a model training method, an apparatus, a device and a storage medium. In the query method, when processing a query request, an interactive operation is performed according to the multimodal description features of the object to be queried to obtain an interactive feature. Based on the interactive features of the object to be queried and the interactive features of the object to be matched, the matching degree of the object to be queried and the object to be matched is calculated, and based on the matching degree, a target object matching the object to be queried is determined from the objects to be matched. In this embodiment, the descriptive features of multiple modes are used, and the descriptive features of multiple modes are interactively operated, so that the descriptive features of different modes and the interactive relationships between them can work together in the process of object recognition, thereby reducing or eliminating the ambiguity of the features of a single mode, which is conducive to effectively improving the accuracy of object recognition.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a query method, a model training method, an apparatus, a device and a storage medium. Background Art

[0002] In the field of object recognition, there is a method for object recognition based on text query description information of the object. However, in some scenarios, when the text is ambiguous, it will lead to the inability to accurately recognize the object based on the text query description information.

[0003] For example, in some scenarios, users use ambiguous text to disguise the banned content they publish, and the ambiguous text is not easily identified as banned content. For another example, in other scenarios, when users use ambiguous text to search for products, the ambiguous text causes the returned product search results to be inconsistent with the products that the user wants to search for. Therefore, a new solution needs to be proposed. Summary of the invention

[0004] Various aspects of the present application provide a query method, a model training method, an apparatus, a device and a storage medium to effectively improve the accuracy of object recognition.

[0005] An embodiment of the present application provides a query method, comprising: in response to a query request, obtaining multimodal description features of an object to be queried; performing a feature interaction operation on the multimodal description features of the object to be queried to obtain a first interaction feature; calculating a matching degree between the object to be queried and the at least one object to be matched based on the first interaction feature and the interaction features of each of the at least one object to be matched; and determining a target object that is compatible with the object to be queried from the at least one object to be matched based on the matching degree.

[0006] The embodiment of the present application also provides a model training method, including: obtaining multimodal description features of the object to be queried according to query description information of the object to be queried; performing feature interaction operations on the multimodal description features of the object to be queried and the multimodal representation features of the object to be matched, respectively, to obtain a first interaction feature and a second interaction feature; calculating the matching degree of the first interaction feature and the second interaction feature according to the model parameters of the query link model; updating the model parameters according to the matching degree and the true value of the matching degree between the object to be queried and the object to be matched, so as to optimize the query link model.

[0007] The embodiment of the present application also provides a query device, including: a first feature acquisition module, used to: respond to a query request and obtain a multimodal description feature of an object to be queried; a first interaction module, used to: perform a feature interaction operation on the multimodal description feature of the object to be queried to obtain a first interaction feature; a matching degree calculation module, used to: calculate the matching degree between the object to be queried and the at least one object to be matched based on the first interaction feature and the interaction features of each of the at least one object to be matched; a target object determination module, used to: determine a target object that is compatible with the object to be queried from the at least one object to be matched based on the matching degree.

[0008] The embodiment of the present application also provides a model training device, including: a first feature acquisition module, used to: acquire the multimodal description features of the object to be queried according to the query description information of the object to be queried; a first interaction module, used to: perform feature interaction operations on the multimodal description features of the object to be queried and the multimodal representation features of the object to be matched, respectively, to obtain a first interaction feature and a second interaction feature; a matching degree calculation module, used to: calculate the matching degree of the first interaction feature and the second interaction feature according to the model parameters of the query link model; a parameter optimization module, used to: update the model parameters according to the matching degree and the true value of the matching degree between the object to be queried and the object to be matched, so as to optimize the query link model.

[0009] An embodiment of the present application also provides an electronic device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the steps in the query method or model training method provided in the embodiment of the present application.

[0010] The embodiments of the present application also provide a computer-readable storage medium storing a computer program, which, when executed, can implement the steps in the query method or model training method provided in the embodiments of the present application.

[0011] In the query method provided in the embodiment of the present application, when processing a query request, an interactive operation is performed according to the multimodal description features of the object to be queried to obtain interactive features. Based on the interactive features of the object to be queried and the interactive features of the object to be matched, the matching degree of the object to be queried and the object to be matched is calculated, and based on the matching degree, the target object that matches the object to be queried is determined from the objects to be matched. In this embodiment, the descriptive features of multiple modes are used, and the descriptive features of multiple modes are interactively operated, so that the descriptive features of different modes and the interactive relationships between them can work together in the process of object recognition, thereby reducing or eliminating the ambiguity of the features of a single mode, which is conducive to effectively improving the accuracy of object recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0013] Figure 1 A flowchart of a query method provided for an exemplary embodiment of the present application;

[0014] Figure 2a A flowchart of a query method provided for another exemplary embodiment of the present application;

[0015] Figure 2b A schematic diagram of a self-interaction operation provided by an exemplary embodiment of the present application;

[0016] Figure 2c A schematic diagram of a two-way interactive operation provided by an exemplary embodiment of the present application;

[0017] Figure 2d A schematic diagram of a two-way interactive operation provided by another exemplary embodiment of the present application;

[0018] Figure 2e A schematic diagram of a commodity query scenario provided by an exemplary embodiment of the present application;

[0019] Figure 3 A flowchart of a model training method provided for an exemplary embodiment of the present application;

[0020] Figure 4 A schematic diagram of the structure of a query device provided by an exemplary embodiment of the present application;

[0021] Figure 5 A schematic diagram of the structure of a model training device provided by an exemplary embodiment of the present application;

[0022] Figure 6 A schematic diagram of the structure of a query device provided by an exemplary embodiment of the present application;

[0023] Figure 7 A schematic diagram of the structure of a model training device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0025] In the prior art, when a user initiates an object query, if the query description information provided is ambiguous, the object to be queried cannot be accurately identified based on the query description information. In view of this technical problem, a solution is provided in some embodiments of the present application. The technical solutions provided in each embodiment of the present application will be described in detail below in conjunction with the accompanying drawings.

[0026] Figure 1 A flowchart of a query method provided by an exemplary embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0027] Step 101: In response to a query request, obtain multimodal description features of the object to be queried.

[0028] Step 102: Perform a feature interaction operation on the multimodal description feature to obtain a first interaction feature.

[0029] Step 103: Calculate the matching degree between the object to be queried and the at least one object to be matched according to the first interaction feature and the interaction features of the at least one object to be matched.

[0030] Step 104: Determine a target object that matches the object to be queried from the at least one object to be matched according to the matching degree.

[0031] Among them, the object can be realized as an objective entity, such as a commodity, an animal, a plant, a building, etc.; the object can also be realized as abstract information, such as a mathematical formula, a concept definition, a literary work, etc., which is not limited in this embodiment.

[0032] Based on different implementation forms of objects, this embodiment can be applied to a variety of different object query scenarios: for example, entity query scenarios such as product query scenarios, animal query scenarios, plant query scenarios, or information query scenarios such as mathematical formula query scenarios, concept query scenarios, and prohibited information query scenarios.

[0033] When a user has an object query requirement, he or she can enter query description information. Based on the query description information, an object matching the query description information can be selected from known objects. For ease of description and distinction, the object described by the query description information entered by the user is called the object to be queried, and the known object in the pre-established query knowledge base is called the object to be matched.

[0034] Modality can refer to different sources of information or different forms of information expression. For example, when information is collected by different sensors such as radar, infrared sensor, camera, etc., the information collected by radar, infrared sensor, camera can be called information of different modalities. For example, when information is expressed in different forms such as images, videos, audio, semantics, etc., the information expressed in the form of images, videos, audio, semantics, etc. can be called information of multiple different modalities.

[0035] Acquiring multimodal description features of the object to be queried includes: acquiring features from different information sources for describing the object to be queried; or acquiring features in different forms for describing the object to be queried.

[0036] Among them, the first interactive feature is obtained according to the feature interaction operation between the multimodal descriptive features of the object to be queried. The feature interaction operation may include cross-calculation and combination calculation between features of multiple different modes. Based on the interactive operation between the features of multiple modes, the mutual influence, combination and interaction between the descriptive features of different modes can be effectively realized, thereby reducing or eliminating the ambiguity of the features of a single mode.

[0037] Among them, the at least one object to be matched may include one object to be matched or multiple objects to be matched, which is not limited in this embodiment. Among them, each object to be matched corresponds to an interaction feature. The calculation method of the interaction feature of the object to be matched is the same or similar to the calculation method of the first interaction feature, which is not repeated here. It should be understood that in this embodiment, the use of "first" to describe the interaction feature is only for the convenience of distinction, and does not constitute a limitation on other information such as the order, quantity or level of the interaction feature.

[0038] Based on the first interaction feature of the object to be queried and the interaction features of the at least one object to be matched, the degree of matching between the object to be queried and the at least one object to be matched can be calculated. For example, for any object to be matched, the degree of matching between the object to be matched and the object to be queried can be obtained based on the degree of matching between the interaction feature of the object to be matched and the first interaction feature of the object to be queried. Based on the at least one calculated degree of matching, a target object that is compatible with the object to be queried can be determined from the at least one object to be matched. For example, the object to be matched with the highest degree of matching can be selected as the target object, or the object to be matched with a degree of matching greater than a set threshold can be selected as the target object, and this embodiment does not impose any restrictions.

[0039] In this embodiment, when processing a query request, an interactive operation is performed based on the multimodal description features of the object to be queried to obtain interactive features. Based on the interactive features of the object to be queried and the interactive features of the object to be matched, the matching degree of the object to be queried and the object to be matched is calculated, and based on the matching degree, the target object that matches the object to be queried is determined from the objects to be matched. In this embodiment, the descriptive features of multiple modes are used, and the descriptive features of multiple modes are interactively operated, so that the descriptive features of different modes and the interactive relationships between them can work together in the process of object recognition, thereby reducing or eliminating the ambiguity of the features of a single mode, which is conducive to effectively improving the accuracy of object recognition.

[0040] Figure 2a A flowchart of a query method provided for another exemplary embodiment of the present application is shown in FIG2 . The method includes:

[0041] Step 201: Respond to a query request and obtain first query description information provided by a user.

[0042] Step 202: Provide the user with at least one other description information that interacts with the first query description information.

[0043] Step 203: In response to the user's selection operation on the at least one other description information, the selected description information is obtained as the second query description information.

[0044] Step 204: input the first query description information and the second query description information into the query link model.

[0045] Step 205: In the query link model, according to the first query description information and the second query description information, respectively extract the description features of the first modality and the description features of the second modality as multimodal description features of the object to be queried.

[0046] Step 206: At the first interaction layer in the query link model, a feature interaction operation is performed on the multimodal description feature to obtain a first interaction feature.

[0047] Step 207: In the second interaction layer in the query link model, a feature interaction operation is performed on the first interaction feature and the interaction feature of at least one object to be matched, to obtain at least one third interaction feature; wherein the interaction feature of the object to be matched is obtained by performing a feature interaction operation on the multimodal representation feature of the object to be matched.

[0048] Step 208: In the scoring layer of the query link model, score the at least one third interaction feature according to the parameters of the scoring layer to obtain the matching degree between the first interaction feature and the interaction feature of the at least one to-be-matched object.

[0049] Step 209: At the output layer of the query link model, a target object matching the object to be queried is determined from at least one object to be matched according to the at least one matching degree, and the target object is output.

[0050] In step 201, the query request may be optionally issued by a user through a terminal device. For example, the user may enter a query request on a query page provided by the terminal device, or enter a voice query instruction through a voice input device provided by the terminal device, which is not limited in this embodiment.

[0051] The first query description information is the query description information provided by the user. In some cases, the first query description information contains a relatively single feature. If the single feature is ambiguous, the response result to the query request will be inaccurate. To solve this problem, the next step 202 can be performed to provide other description information to the user for the user to select.

[0052] In step 202, optionally, the method of providing the at least one other description information to the user may include: displaying the at least one other description information through a display screen provided by the terminal, or playing the at least one other description information through voice broadcast. This embodiment is not limited.

[0053] Among them, the first query description information and the at least one other description information have an interactive relationship, and the interactive relationship means that: the at least one other description information and the first query description information can describe the same object, and the at least one other description information can help reduce or eliminate the ambiguity of the first query description information; accordingly, the first query description information can also help reduce or eliminate the ambiguity of the at least one other description information.

[0054] For example, when a user initiates a search query operation, the first query description information entered is the text description information of "an animal that picks leaves from trees and eats them". Since there are many animals that can pick leaves and eat them directly, it is impossible to accurately return search results to the user. Based on this step, other query description information can be provided to the user, such as pictures of koalas, monkeys, or giraffes. The above pictures can effectively help reduce or eliminate the ambiguity of the first query description information.

[0055] The at least one other description information may be obtained by using the following implementation method:

[0056] Optionally, multiple objects to be matched that are adapted to the first query description information may be obtained from the multimodal knowledge graph.

[0057] Among them, the multimodal knowledge graph is pre-established. In the multimodal knowledge graph, the full amount of objects to be matched, the multimodal representation information of the objects to be matched, and the correspondence between the objects to be matched and their multimodal representation information are saved, which realizes the unified representation of the objects to be matched at the knowledge level and lays the foundation for reducing or eliminating ambiguity in the object recognition process.

[0058] Based on this, the first query description information and the representation information in the multimodal knowledge graph can be matched. For example, if the first query description information is in text format, the first query description information and the representation information in the multimodal knowledge graph can be text-matched. If there is representation information that matches the first query description information, the object to be matched corresponding to the representation information can be used as the object to be matched adapted to the first query description information. Usually, when the first query description information is ambiguous, the number of objects to be matched adapted to it is multiple.

[0059] Next, from the multimodal representation information corresponding to each of the plurality of objects to be matched, other representation information belonging to a different modality from the first query description information is selected as the at least one other description information. After the user obtains the at least one other description information, the user can select description information matching the query requirement from the at least one other description information according to the query requirement.

[0060] Continuing with the above example, the first query description information is: an animal that picks leaves from trees and eats them. If in the multimodal knowledge graph, there is representation information that is adapted to "picking leaves from trees and eating" in the representation information of multiple animals such as koalas, monkeys, and giraffes, the representation information of other modalities of koalas, monkeys, and giraffes can be obtained respectively. For example, pictures of koalas, monkeys, and giraffes can be obtained and displayed to users. If the user actually wants to query koalas, he can select pictures of koalas from the above pictures.

[0061] In step 203, based on the user's selection operation, other description information selected by the user may be used as second query description information.

[0062] In some preferred embodiments, the first query description information includes: text description information; the second query description information includes: image description information. Based on this, the image description information can be fully utilized to supplement the text description information to reduce or eliminate the ambiguity of the text description information. For example, continuing the above example, if the user selects a picture of a koala, the picture of a koala can be used as the second query description information.

[0063] Next, in step 204, the first query description information and the second query description information may be input into a query link model. The query link model is pre-trained and used to identify multi-modal features contained in the query description information and to link the query result to one or more objects. The following will exemplarily illustrate the internal working mode of the query link model in conjunction with the subsequent steps.

[0064] In step 205, after the query link model obtains the input first query description information and second query description information, it can respectively perform feature extraction on the two query description information to obtain description features of the first modality and description features of the second modality.

[0065] The description features of the first mode and the description features of the second mode may respectively include one or more different description features, which is not limited in this embodiment.

[0066] Optionally, if the query description information is information in text format, semantic feature extraction can be performed on the information in text format. Among them, the semantic feature extraction operation can be implemented based on the word2vec (word to vector) model, or can be implemented based on the ELMo model (a deep contextualized word representation model), which is not limited in this embodiment. For example, based on the word2vec (word to vector) model, semantic feature extraction can be performed on the name of the object to be queried contained in the query description information to obtain the semantic features of the name of the object to be queried. Alternatively, based on the ELMo model, semantic feature extraction can be performed on the context information of the name of the object to be queried to obtain the contextual semantic features of the name of the object to be queried.

[0067] Optionally, if the query description information is information in a picture format, image features may be extracted from the picture format information. The method of extracting image features may be based on at least one of VGGNet (Visual Geometry Group Network, a neural network model), AlexNet (a neural network model), and InceptionNet (a neural network model), and this embodiment does not limit this.

[0068] Optionally, if the query description information is in audio format, speech recognition can be performed on the audio format information, and semantic features can be extracted based on the speech recognition result. Alternatively, audio features can be directly extracted from the audio format information, which is not limited in this embodiment.

[0069] When the query description information is in other formats, feature extraction can be performed using feature extraction methods corresponding to information in other formats, which will not be described in detail.

[0070] After the multimodal description features of the object to be queried are obtained, step 206 may be performed next to perform a feature interaction operation on the multimodal description features of the object to be queried to obtain a first interaction feature.

[0071] Optionally, the multimodal description features of the object to be queried may include: at least one of the semantic features of the name of the object to be queried, the contextual semantic features of the name of the object to be queried, and the image features of the object to be queried. In step 206, optionally, for ease of description and distinction, the computational layer in the query link model for performing feature interaction operations on the multimodal description features may be recorded as the first interaction layer.

[0072] Optionally, in the first interaction layer of the query link model, the multimodal description features of the query object may be self-interactively processed based on an attention mechanism to obtain the self-interaction feature vectors of the multimodal description features of the query object.

[0073] Optionally, the following will take the description feature of any modality in the multimodal description features of the object to be queried as an example to exemplify the self-interaction operation based on the attention mechanism. For the convenience of description, the description feature of any modality is recorded as the first description feature.

[0074] Optionally, for the first descriptive feature, the similarity between the first descriptive feature and each feature in the multimodal descriptive feature of the object to be queried can be calculated to obtain multiple self-interaction weights corresponding to the first descriptive feature; then, based on the multiple self-interaction weights, the multimodal descriptive features of the object to be queried can be weightedly calculated to obtain the self-interaction vector of the first descriptive feature.

[0075] The following will be combined Figure 2b , which is further explained with a specific example.

[0076] Assumption: The descriptive features of the object to be queried include: descriptive features X1, X2, and X3. Figure 2bAs shown, when the description feature X1 is self-interactively calculated, the similarity between the description feature X1 and the description feature X2 can be calculated respectively: S11 = S(X1, X2), the similarity between the description feature X1 and the description feature X2: S12 = S(X1, X2) and the similarity between the description feature X1 and the description feature X3: S13 = S(X1, X3).

[0077] After obtaining S11, S12 and S13, they can be normalized to obtain three corresponding self-attention weights A11, A12 and A13. Then, according to X1, X2, X3 and A11, A12, A13, weighted calculation is performed to obtain the self-interaction vector X1`=X1*A1+X2*A2+X3*A3 corresponding to X1.

[0078] Accordingly, the above method can be used to calculate the self-interaction vectors of the description features of other modes in the multimodal description features of the query object. Figure 2b The illustrated calculation process obtains the self-interaction vector X2' corresponding to the description feature X2 and the self-interaction vector X3' corresponding to the description feature X3.

[0079] Next, the self-interaction feature vectors of the multimodal description features of the query object can be fused to obtain a first interaction feature. Optionally, the fusion operation can be implemented as a vector concatenation operation. For example, when the self-interaction vectors of the multimodal description features are X1`, X2`, and X3`, respectively, the first interaction feature can be implemented as: (X1`, X2`, X3`).

[0080] In the next step 207, feature interaction operations may be performed on the first interaction feature and the interaction features of the at least one object to be matched. The interaction features of each object to be matched are obtained by performing feature interaction operations on the multimodal representation features of the object to be matched. For optional implementations of performing feature interaction operations on the multimodal representation features of the object to be matched, reference may be made to the optional implementations of performing interaction operations on the multimodal description features of the query object described above, which will not be described in detail here.

[0081] It is worth noting that the multimodal knowledge graph pre-constructed in the embodiment of the present application contains multimodal representation features of all objects to be matched. In some optional embodiments, the interaction features of each object to be matched can be calculated in advance based on the multimodal representation features of each object to be matched, and the interaction features of each object to be matched can be stored in the multimodal knowledge graph for standby use. In other optional embodiments, when calculating the first interaction feature, the multimodal representation features of each object to be matched can be obtained from the multimodal knowledge graph, and the interaction features of each object to be matched can be calculated in real time in the first interaction layer of the query link model, which is not limited in this embodiment.

[0082] Optionally, for any object to be matched, the multimodal representation features of the object to be matched may include: at least one of: structural features of the object to be matched, image features of the object to be matched, semantic features of the name of the object to be matched, and summary features of the object to be matched.

[0083] The following will take any one of the at least one to-be-matched object as an example for exemplary description. For ease of description, the interactive object of the to-be-matched object is marked as a second interactive feature.

[0084] Optionally, the operation of performing a feature interaction operation on the first interaction feature and the second interaction feature to obtain a third interaction feature may be implemented at a second interaction layer in the query link model.

[0085] Optionally, in the second interaction layer, based on the attention mechanism, bidirectional interaction processing may be performed on the first interaction feature and the second interaction feature to obtain a bidirectional interaction feature vector of the object to be queried and a bidirectional interaction feature vector of the object to be matched.

[0086] Optionally, based on the above-mentioned embodiment, it can be known that the first interaction feature includes multiple feature vectors corresponding to the multimodal description features of the object to be queried, and the second interaction feature includes multiple feature vectors corresponding to the multimodal representation features of the object to be matched.

[0087] The following will take any feature vector included in the first interaction feature as an example to exemplify the bidirectional interaction operation based on the attention mechanism. For the convenience of description, the any feature vector is recorded as the first feature vector.

[0088] Optionally, for the first feature vector, the similarity between the first feature vector and multiple feature vectors in the second interaction feature can be calculated to obtain multiple bidirectional interaction weights; then, based on the multiple bidirectional interaction weights, multiple features in the second interaction feature can be weighted to obtain the bidirectional interaction vector corresponding to the first feature vector. Figure 2c and Figure 2d , which is further explained with a specific example.

[0089] Assume that the first interactive features include: feature vectors X1`, X2`, X3`, and the second interactive features include: feature vectors Y1`, Y2`, Y3`, Y4`. Figure 2cAs shown, when the feature vector X1` is interactively calculated according to the feature vector included in the second interactive feature, the similarity between the feature vector X1` and the feature vectors Y1`, Y2`, Y3`, and Y4` can be calculated respectively: S11`=S(X1`, Y1`), the similarity between the feature vector X1` and the feature vector Y2`: S12`=S(X1`, Y2`), the similarity between the feature vector X1` and the feature vector Y3`: S13`=S(X1`, Y3`), and the similarity between the feature vector X1` and the feature vector Y4`: S14`=S(X1`, Y4`).

[0090] After obtaining S11`, S12`, S13` and S14`, they can be normalized to obtain three corresponding self-attention weights A11`, A12`, A13` and A14`. Then, according to Y1`, Y2`, Y3`, Y4` and A11`, A12`, A13` and A14`, weighted calculation is performed to obtain the bidirectional interaction vector X1` corresponding to X1` = Y1`*A1`+Y2`*A2`+Y3`*A3`+Y4`*A4`.

[0091] Accordingly, the above method is adopted, combined with Figure 2c , calculate the bidirectional interaction vectors corresponding to other feature vectors in the first interaction feature.

[0092] Next, the bidirectional interaction vectors corresponding to each feature vector in the first interaction feature can be fused to obtain the bidirectional interaction feature vector of the object to be queried. Optionally, the fusion operation can be implemented as a vector concatenation operation. For example, when the bidirectional interaction vectors corresponding to each feature vector in the first interaction feature are X1``, X2``, and X3`` respectively, the bidirectional interaction feature vector of the object to be queried can be implemented as: (X1``, X2``, X3``).

[0093] In step 207, the bidirectional interaction feature vector of the object to be matched can be obtained by the method described above, which will not be repeated here. For example, continuing the above example, the calculated bidirectional interaction feature vector of the object to be matched can be implemented as: (Y1``, Y2``, Y3``, Y4``).

[0094] Optionally, the bidirectional interaction feature vector of the object to be queried and the bidirectional interaction feature vector of the object to be matched may be merged to obtain the third interaction feature, which may include: concatenating the bidirectional interaction feature vector of the object to be queried and the bidirectional interaction feature vector of the object to be matched, such as Figure 2d shown.

[0095] For example, following the above example, the third interaction feature can be implemented as (X1``, X2``, X3``, Y1``, Y2``, Y3``, Y4``).

[0096] Based on the above steps, feature interaction operations may be performed on the first interaction feature and the interaction features corresponding to the at least one to-be-matched object, respectively, to obtain at least one third interaction feature.

[0097] After obtaining at least one third feature, steps 208 and 209 may be executed. In the scoring layer of the query link model, the at least one third interactive feature is scored respectively according to the parameters of the scoring layer to obtain the matching degree between the first interactive feature and the interactive feature of the at least one object to be matched. In the output layer of the query link model, according to the calculated at least one matching degree, a target object that matches the object to be queried is determined from the at least one object to be matched, and the target object is output. The target object may be returned to the user as a query result of the query request.

[0098] In this embodiment, the description features of multiple modes of the query object are interactively operated to obtain interactive features, and the object is identified based on the interactive features, so that the description features of different modes and the interactive relationship between them can work together in the object identification process, thereby reducing or eliminating the ambiguity of the features of a single mode, which is conducive to effectively improving the accuracy of object identification. In addition, the interactive features of the query object and the interactive features of the matching object are further interacted, which is conducive to improving the accuracy and reliability of the object recognition results.

[0099] The above or below embodiments of the present application are applicable to a variety of different object recognition scenarios. An exemplary description will be given below.

[0100] A typical application scenario is the product query scenario. When a user searches for a product on an e-commerce platform displayed on a terminal, he or she can enter the product's query description information. For example, Figure 2e The query description information in text format shown is: a kind of warm shoes. The terminal can send the description information to the server, and the server provides the query results. In the actual commodity category, there are many kinds of warm shoes, such as cotton slippers, snow boots, velvet boots, electric heating shoes, etc. When there is ambiguity in the query description information in text format, the server cannot provide users with more accurate search results. Next, the server can obtain other modal description information of the above-mentioned multiple shoe commodities from the pre-established commodity multimodal knowledge graph, such as image description information or commodity details description information. Figure 2e In the example, the image description information of the shoe product is used for illustration. The server can send the images of the above-mentioned various shoe products to the terminal for display.

[0101] After viewing multiple images of shoe products through the terminal, the user can select one or more images that meet the needs from the multiple images of shoe products. Figure 2eAs shown, the user selects an image of a cotton slipper from multiple images of shoe products. After the user selects, the terminal can send the image of the cotton slipper to the server. Next, the server can use the above text format query description information and the image of the cotton slipper as multimodal description information of the product to be queried, and calculate the matching products that are compatible with the product to be queried based on the query methods provided in the above embodiments. Figure 2e The query results of various cotton slippers returned by the server are shown. Based on this implementation, when the product query description information input by the user is ambiguous, the product search results can be accurately returned to the user, thereby improving the query efficiency and the order conversion rate of the e-commerce platform.

[0102] Another typical application scenario is: prohibited content retrieval scenario. Existing prohibited content retrieval methods usually rely on the analysis of information in text format. Therefore, when some information publishers publish prohibited content, they will publish content with ambiguous text format to avoid the risk of violation. At the same time, in order to facilitate the audience of the content to understand the prohibited content, the information publisher also publishes pictures to clearly display the prohibited content in the pictures. The above-mentioned information publishing method makes it impossible to accurately screen out prohibited content. Based on the query method provided in this embodiment, suspicious prohibited content in text format and the corresponding pictures can be used as multimodal description features of the prohibited content to be retrieved, so as to accurately identify prohibited content and purify the network environment.

[0103] The query link model provided in the above embodiment can be trained using the following optional implementation methods, which will be described below with reference to the accompanying drawings.

[0104] Figure 3 A flow chart of a model training method provided for an exemplary embodiment of the present application is shown in FIG. Figure 3 As shown, the method includes:

[0105] Step 301: Acquire multimodal description features of the object to be queried according to the query description information of the object to be queried.

[0106] Step 302: Perform feature interaction operations on the multimodal description features of the query object and the multimodal representation features of the object to be matched, respectively, to obtain a first interaction feature and a second interaction feature.

[0107] Step 303: Calculate the matching degree of the first interaction feature and the second interaction feature according to the model parameters of the query link model.

[0108] Step 304: Update the model parameters of the query link model according to the matching degree and the true value of the matching degree between the object to be queried and the object to be matched, so as to optimize the query link model.

[0109] The query description information of the object to be queried can be obtained based on the user's historical query records. For example, the query corpus input by the user when initiating a historical query operation can be obtained as the query description information. The query description information is used as a training sample of the query link model to optimize the model parameters of the query link model.

[0110] In some exemplary embodiments, the multimodal description features of the object to be queried may include: at least one of the semantic features of the name of the object to be queried, the contextual semantic features of the name of the object to be queried, and the image features of the object to be queried. For optional implementations of obtaining the multimodal description features of the object to be queried, reference may be made to the description of the aforementioned embodiments, which will not be repeated here.

[0111] In some exemplary embodiments, the multimodal representation features of the object to be matched may include: at least one of: structural features of the object to be matched, image features of the object to be matched, semantic features of the name of the object to be matched, and summary features of the object to be matched.

[0112] The multimodal representation features of the objects to be matched can be stored in a multimodal knowledge graph. When constructing a multimodal knowledge graph, at least one of the following methods can be executed to obtain the multimodal representation features of the objects to be matched:

[0113] Method 1: Generate structural features of the object to be matched based on the object to be matched and other objects that have a set relationship.

[0114] Optionally, for each object to be matched in the data set, a triple data can be constructed, which consists of a head object h, a relationship r, and a tail object t, and is formally expressed as: (h, r, t). Then, based on the Translating Embedding (TransE) algorithm, other objects that have a set relationship with the object to be matched can be screened from the data set. That is, the TransE algorithm is used to find other objects h and relationships r to be matched that can satisfy h+r≈t. Then, the vector consisting of the object to be matched, the set relationship, and other objects can be used as the structural vector of the object to be matched.

[0115] When calculating other objects that have a set relationship with the object to be matched based on TransE, the loss function can be set to a hinge loss function. The hinge loss function can be implemented as:

[0116] L=∑∑max(γ+E(h,r,t)-E(h',r,t'),0)

[0117] Where γ is the margin of the hinge loss function, E(h,r,t) is a positive example, and E(h',r,t') is a negative example. Negative examples can be generated by randomly replacing the head object and / or the tail object.

[0118] For example, when the triple data corresponding to the object to be matched is (h, r, t), when the head object and / or the tail object are randomly replaced, the following momentum functions can be generated:

[0119] E1=||h`+rt`||

[0120] E2=||h+rt`||

[0121] E3=||h+rt||

[0122] E4=||h`+rt||

[0123] E5=||(h`+h)+r-(t`+t)||

[0124] Among them, h` represents the randomly replaced head object, and t` represents the randomly replaced tail object.

[0125] The E(h', r, t') in the above hinge loss function can be implemented as any one or more combinations of the above E1-E5, which is not limited in this embodiment. In some embodiments, E(h', r, t') can be implemented as the sum of E1-E5, that is:

[0126] Method 2: The image features of the object to be matched can be calculated based on the image of the object to be matched. The operation of calculating the image features of the object to be matched can be implemented based on at least one model of VGGNet, AlexNet InceptionNet, and will not be described in detail.

[0127] It should be understood that each object to be matched may correspond to multiple images, and based on the multiple images, multiple image feature vectors of the object to be matched may be calculated. Then, the cosine similarity between the image feature vectors may be calculated, and an image feature similarity network may be constructed based on the calculated cosine similarity values. In the image feature similarity network, a node represents an image feature vector, and an edge in the network represents the cosine similarity value between the two nodes connected by the edge.

[0128] Next, the PageRank algorithm can be used to calculate the PageRank value of each node in the image feature vector similarity network, and several nodes with the highest PageRank values ​​are selected as the image feature vectors of the object to be matched. Optionally, in some embodiments, these several image feature vectors can be further averaged or maximized to obtain the aggregated image feature vector of the object to be matched.

[0129] Method 3: Generate a word vector based on the name of the object to be matched to obtain the semantic features of the name of the object to be matched.

[0130] Method 4: Generate a word vector based on the summary information of the object to be matched to obtain the summary features of the object to be matched.

[0131] In method 3 and method 4, the operation of generating word vectors can be implemented based on the word2vec model or the ELMo model, which will not be described here.

[0132] In some exemplary embodiments, a method of performing feature interaction operation on the multimodal description features of the object to be queried to obtain a first interaction feature includes: in the first interaction layer of the query link model, based on the attention mechanism, performing self-interaction processing on the multimodal description features of the object to be queried to obtain the self-interaction feature vectors of the multimodal description features of the object to be queried; fusing the self-interaction feature vectors of the multimodal description features of the object to be queried to obtain the first interaction feature. In some exemplary embodiments, a method of performing self-interaction processing on the multimodal description features of the object to be queried to obtain the self-interaction feature vectors of the multimodal description features of the object to be queried based on the attention mechanism includes: for the description features of any mode in the multimodal description features of the object to be queried, calculating the similarity between the description features and each feature in the multimodal description features of the object to be queried to obtain multiple self-interaction weights corresponding to the description features; according to the multiple self-interaction weights, performing weighted calculation on the multimodal description features of the object to be queried to obtain the self-interaction vector of the description features.

[0133] In some exemplary embodiments, a method for calculating the degree of match between the first interaction feature and the second interaction feature based on model parameters of a query link model includes: in the query link model, performing a feature interaction operation on the first interaction feature and the second interaction feature to obtain a third interaction feature; in a scoring layer of the query link model, scoring the third interaction feature according to the parameters of the scoring layer to obtain the degree of match between the first interaction feature and the second interaction feature.

[0134] In some exemplary embodiments, a method for performing feature interaction operations on the first interaction feature and the second interaction feature to obtain a third interaction feature includes: in the second interaction layer in the query link model, based on the attention mechanism, performing bidirectional interaction processing on the first interaction feature and the second interaction feature to obtain a bidirectional interaction feature vector of the object to be queried and a bidirectional interaction feature vector of the object to be matched; and fusing the bidirectional interaction feature vector of the object to be queried and the bidirectional interaction feature vector of the object to be matched to obtain the third interaction feature.

[0135] In some exemplary embodiments, the first interaction feature includes multiple feature vectors corresponding to the multimodal description features of the object to be queried, and the second interaction feature includes multiple feature vectors corresponding to the multimodal representation features of the object to be matched; accordingly, based on the attention mechanism, bidirectional interaction processing is performed on the first interaction feature and the second interaction feature to obtain a bidirectional interaction feature vector of the object to be queried, including: for any feature vector in the first interaction feature, calculating the similarity between the feature vector and multiple feature vectors in the second interaction feature to obtain multiple bidirectional interaction weights; according to the multiple bidirectional interaction weights, performing weighted calculation on multiple features in the second interaction feature to obtain the bidirectional interaction vector of the feature; splicing the bidirectional interaction vectors of the multiple features contained in the first interaction feature to obtain the bidirectional interaction feature vector of the object to be queried.

[0136] In this embodiment, by constructing a multimodal knowledge graph, the representation of the objects to be matched at the knowledge level can be unified, which is conducive to the mutual influence and combination of multimodal representation features. Based on the multimodal features of the objects to be queried and the objects to be matched, the interactive features are calculated, and the query link model is trained based on the calculated interactive features, so that the query link model can continuously and automatically learn the interactive relationship between the multimodal features, and automatically learn the relationship between the interactive features of the objects to be queried and the interactive features of the objects to be matched, which is conducive to accurately identifying the objects that match the query request based on the multimodal features with interactive relationships in the actual application of the model.

[0137] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 201 to 204 can be device A; for another example, the execution subject of steps 201 and 202 can be device A, and the execution subject of step 203 can be device B; and so on.

[0138] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 201, 202, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0139] Figure 4 is a schematic diagram of the structure of a query device provided by an exemplary embodiment of the present application. Figure 4 As shown, the device comprises:

[0140] The first feature acquisition module 401 is used to respond to a query request and acquire multimodal description features of the object to be queried.

[0141] The first interaction module 402 is used to perform a feature interaction operation on the multimodal description feature of the object to be queried to obtain a first interaction feature.

[0142] The matching degree calculation module 403 is used to calculate the matching degree between the object to be queried and the at least one object to be matched according to the first interaction feature and the interaction features of the at least one object to be matched.

[0143] The target object determination module 404 is used to determine, according to the matching degree, a target object that matches the object to be queried from the at least one object to be matched.

[0144] In some exemplary embodiments, when the first feature acquisition module 401 responds to a query request and acquires multimodal description features of an object to be queried, it is specifically used to: respond to the query request and acquire first query description information provided by a user; provide the user with at least one other description information that has an interactive relationship with the first query description information; respond to the user's selection operation from at least one other description information and acquire the selected description information as second query description information; and acquire description features of the first modality and description features of the second modality from the first query description information and the second query description information, respectively, as multimodal description features of the object to be queried.

[0145] In some exemplary embodiments, the first feature acquisition module 401 is also used to: obtain, from a multimodal knowledge graph, a plurality of objects to be matched that are adapted to the first query description information; and select, from the multimodal representation information corresponding to each of the plurality of objects to be matched, other representation information belonging to a different modality from the first query description information as the at least one other description information.

[0146] In some exemplary embodiments, the first query description information includes: text description information; the second query description information includes: image description information.

[0147] In some exemplary embodiments, when the first interaction module 402 performs a feature interaction operation on the multimodal description features of the object to be queried to obtain the first interaction feature, it is specifically used to: at the first interaction layer of the query link model, based on the attention mechanism, perform self-interaction processing on the multimodal description features of the object to be queried to obtain the self-interaction feature vectors of the multimodal description features of the object to be queried; and fuse the self-interaction feature vectors of the multimodal description features of the object to be queried to obtain the first interaction feature.

[0148] In some exemplary embodiments, when the first interaction module 402 performs self-interaction processing on the multimodal description features of the object to be queried based on the attention mechanism to obtain the self-interaction feature vectors of the multimodal description features of the object to be queried, it is specifically used to: calculate the similarity between the description features of any modality in the multimodal description features of the object to be queried and each feature in the multimodal description features of the object to be queried, and obtain multiple self-interaction weights corresponding to the description features; and perform weighted calculation on the multimodal description features of the object to be queried according to the multiple self-interaction weights to obtain the self-interaction vector of the description features.

[0149] In some exemplary embodiments, the first interaction module 402 is further used to: for any one of the at least one object to be matched, perform a feature interaction operation on the multimodal representation feature of the object to be matched to obtain an interaction feature of the object to be matched as a second interaction feature.

[0150] In some exemplary embodiments, the matching degree calculation module 403 includes: a second interaction module 4031 and a scoring module 4032; when the matching degree calculation module 403 calculates the matching degree between the object to be queried and the at least one object to be matched based on the first interaction feature and the respective interaction features of at least one object to be matched, it is specifically used to: perform feature interaction operations on the first interaction feature and the second interaction feature in the query link model through the second interaction module 4031 to obtain a third interaction feature; and score the third interaction feature in the scoring layer of the query link model according to the parameters of the scoring layer through the scoring module 4032 to obtain the matching degree between the object to be queried and the object to be matched.

[0151] In some exemplary embodiments, the second interaction module 4031 performs feature interaction operations on the first interaction feature and the second interaction feature to obtain a third interaction feature, which is specifically used to: in the second interaction layer in the query link model, based on the attention mechanism, perform bidirectional interaction processing on the first interaction feature and the second interaction feature to obtain a bidirectional interaction feature vector of the object to be queried and a bidirectional interaction feature vector of the object to be matched; and fuse the bidirectional interaction feature vector of the object to be queried and the bidirectional interaction feature vector of the object to be matched to obtain the third interaction feature.

[0152] In some exemplary embodiments, the first interaction feature includes multiple feature vectors corresponding to the multimodal description features of the object to be queried, and the second interaction feature includes multiple feature vectors corresponding to the multimodal representation features of the object to be matched; when the second interaction module 4031 performs bidirectional interaction processing on the first interaction feature and the second interaction feature based on the attention mechanism to obtain the bidirectional interaction feature vector of the object to be queried, it is specifically used to: for any feature vector in the first interaction feature, calculate the similarity between the feature vector and multiple feature vectors in the second interaction feature to obtain multiple bidirectional interaction weights; according to the multiple bidirectional interaction weights, perform weighted calculation on the multiple feature vectors in the second interaction feature to obtain the bidirectional interaction vector of the feature; splice the bidirectional interaction vectors of the multiple feature vectors contained in the first interaction feature to obtain the bidirectional interaction feature vector of the object to be queried.

[0153] In this embodiment, when processing a query request, an interactive operation is performed based on the multimodal description features of the object to be queried to obtain interactive features. Based on the interactive features of the object to be queried and the interactive features of the object to be matched, the matching degree of the object to be queried and the object to be matched is calculated, and based on the matching degree, the target object that matches the object to be queried is determined from the objects to be matched. In this embodiment, the descriptive features of multiple modes are used, and the descriptive features of multiple modes are interactively operated, so that the descriptive features of different modes and the interactive relationships between them can work together in the process of object recognition, thereby reducing or eliminating the ambiguity of the features of a single mode, which is conducive to effectively improving the accuracy of object recognition.

[0154] Figure 5 is a schematic diagram of the structure of a model training device provided by an exemplary embodiment of the present application. Figure 5 As shown, the device comprises:

[0155] The first feature acquisition module 501 is used to acquire multimodal description features of the object to be queried according to the query description information of the object to be queried.

[0156] The first interaction module 502 is used to perform feature interaction operations on the multimodal description features of the object to be queried and the multimodal representation features of the object to be matched, respectively, to obtain a first interaction feature and a second interaction feature.

[0157] The matching degree calculation module 503 is used to calculate the matching degree between the first interaction feature and the second interaction feature according to the model parameters of the query link model.

[0158] The parameter optimization module 504 is used to update the model parameters according to the matching degree and the true value of the matching degree between the object to be queried and the object to be matched, so as to optimize the query link model.

[0159] In some exemplary embodiments, the multimodal description features of the object to be queried may include: at least one of: semantic features of the name of the object to be queried, contextual semantic features of the name of the object to be queried, and image features of the object to be queried.

[0160] In some exemplary embodiments, the multimodal representation features of the object to be matched may include: at least one of: structural features of the object to be matched, image features of the object to be matched, semantic features of the name of the object to be matched, and summary features of the object to be matched.

[0161] In some exemplary embodiments, the device also includes: a second feature acquisition module 505, used to: perform at least one of the following operations to obtain multimodal representation features of the object to be matched: generate structural features of the object to be matched based on the object to be matched and other objects that have a set relationship; calculate image features of the object to be matched based on the image of the object to be matched; generate a word vector based on the name of the object to be matched to obtain semantic features of the name of the object to be matched; generate a word vector based on the summary information of the object to be matched to obtain summary features of the object to be matched.

[0162] In some exemplary embodiments, when the second feature acquisition module 505 generates the structural features of the object to be matched based on the object to be matched and other objects with a set relationship, it is specifically used to: calculate other objects with a set relationship with the object to be matched based on a translation embedding algorithm; and use a vector composed of the object to be matched, the set relationship and the other objects as the structural vector of the object to be matched.

[0163] In some exemplary embodiments, when the first interaction module 502 performs feature interaction operations on the multimodal description features of the object to be queried to obtain the first interaction feature, it is specifically used to: at the first interaction layer of the query link model, based on the attention mechanism, perform self-interaction processing on the multimodal description features of the object to be queried to obtain the self-interaction feature vectors of the multimodal description features of the object to be queried; and fuse the self-interaction feature vectors of the multimodal description features of the object to be queried to obtain the first interaction feature.

[0164] In some exemplary embodiments, when the first interaction module 502 performs self-interaction processing on the multimodal description features of the object to be queried based on the attention mechanism to obtain the self-interaction feature vectors of the multimodal description features of the object to be queried, it is specifically used to: calculate the similarity between the description features of any modality in the multimodal description features of the object to be queried and each feature in the multimodal description features of the object to be queried, and obtain multiple self-interaction weights corresponding to the description features; and perform weighted calculation on the multimodal description features of the object to be queried according to the multiple self-interaction weights to obtain the self-interaction vector of the description features.

[0165] In some exemplary embodiments, the device also includes: a matching degree calculation module 503 includes: a second interaction module 5031 and a scoring module 5032; the matching degree calculation module 503, when calculating the matching degree of the first interaction feature and the second interaction feature according to the model parameters of the query link model, is specifically used to: through the second interaction module 5031, in the query link model, perform feature interaction operations on the first interaction feature and the second interaction feature to obtain a third interaction feature; through the scoring module 5032, in the scoring layer of the query link model, according to the parameters of the scoring layer, score the third interaction feature to obtain the matching degree of the first interaction feature and the second interaction feature.

[0166] In some exemplary embodiments, when the second interaction module 5031 performs feature interaction operations on the first interaction feature and the second interaction feature to obtain the third interaction feature, it is specifically used to: in the second interaction layer in the query link model, based on the attention mechanism, perform bidirectional interaction processing on the first interaction feature and the second interaction feature to obtain the bidirectional interaction feature vector of the object to be queried and the bidirectional interaction feature vector of the object to be matched; and fuse the bidirectional interaction feature vector of the object to be queried and the bidirectional interaction feature vector of the object to be matched to obtain the third interaction feature.

[0167] In some exemplary embodiments, the first interaction feature includes multiple feature vectors corresponding to the multimodal description features of the object to be queried, and the second interaction feature includes multiple feature vectors corresponding to the multimodal representation features of the object to be matched; accordingly, when the second interaction module 5031 performs bidirectional interaction processing on the first interaction feature and the second interaction feature based on the attention mechanism to obtain the bidirectional interaction feature vector of the object to be queried, it is specifically used to: for any feature vector in the first interaction feature, calculate the similarity between the feature vector and multiple feature vectors in the second interaction feature to obtain multiple bidirectional interaction weights; according to the multiple bidirectional interaction weights, perform weighted calculation on multiple features in the second interaction feature to obtain the bidirectional interaction vector of the feature; splice the bidirectional interaction vectors of each of the multiple features contained in the first interaction feature to obtain the bidirectional interaction feature vector of the object to be queried.

[0168] In this embodiment, the interaction features are calculated based on the multimodal features of the object to be queried and the object to be matched, and the query link model is trained based on the calculated interaction features. This allows the query link model to continuously and automatically learn the interaction relationship between the multimodal features, and automatically learn the relationship between the interaction features of the object to be queried and the interaction features of the object to be matched, which is beneficial in the actual application of the model, and accurately identifies the objects that match the query request based on the multimodal features that have an interactive relationship.

[0169] Figure 6 is a schematic diagram of the structure of a query device provided by an exemplary embodiment of the present application. Figure 6 As shown, the query device includes: a memory 601 and a processor 602 .

[0170] The memory 601 is used to store computer programs and can be configured to store various other data to support operations on the query device. Examples of such data include instructions for any application or method operating on the query device, contact data, phone book data, messages, pictures, videos, etc.

[0171] The processor 602 is coupled to the memory 601 and is used to execute the computer program in the memory 601, so as to: respond to a query request and obtain multimodal description features of the object to be queried; the first interaction module 402 is used to: perform feature interaction operations on the multimodal description features of the object to be queried to obtain a first interaction feature; calculate the matching degree between the object to be queried and the at least one object to be matched according to the first interaction feature and the interaction features of each of the at least one object to be matched; and determine, from the at least one object to be matched, a target object that is adapted to the object to be queried according to the matching degree.

[0172] Further optionally, when the processor 602 responds to a query request and obtains multimodal description features of the object to be queried, it is specifically used to: respond to the query request and obtain first query description information provided by a user; provide the user with at least one other description information that has an interactive relationship with the first query description information; respond to the user's selection operation from at least one other description information, obtain the selected description information as second query description information; and obtain description features of the first modality and description features of the second modality from the first query description information and the second query description information, respectively, as multimodal description features of the object to be queried.

[0173] Further optionally, the processor 602 is also used to: obtain, from the multimodal knowledge graph, a plurality of objects to be matched that are adapted to the first query description information; and select, from the multimodal representation information corresponding to each of the plurality of objects to be matched, other representation information belonging to a different modality from the first query description information as the at least one other description information.

[0174] Further optionally, the first query description information includes: text description information; the second query description information includes: image description information.

[0175] Further optionally, when the processor 602 performs a feature interaction operation on the multimodal description features of the object to be queried to obtain the first interaction feature, it is specifically used to: at the first interaction layer of the query link model, based on the attention mechanism, perform self-interaction processing on the multimodal description features of the object to be queried to obtain the self-interaction feature vectors of the multimodal description features of the object to be queried; and fuse the self-interaction feature vectors of the multimodal description features of the object to be queried to obtain the first interaction feature.

[0176] Further optionally, when the processor 602 performs self-interaction processing on the multimodal description features of the object to be queried based on the attention mechanism to obtain the self-interaction feature vectors of the multimodal description features of the object to be queried, it is specifically used to: calculate the similarity between the description features and each feature in the multimodal description features of the object to be queried for the description features of any modality in the multimodal description features of the object to be queried, and obtain multiple self-interaction weights corresponding to the description features; and perform weighted calculation on the multimodal description features of the object to be queried according to the multiple self-interaction weights to obtain the self-interaction vector of the description features.

[0177] Further optionally, the processor 602 is further configured to: for any one of the at least one object to be matched, perform a feature interaction operation on a multimodal representation feature of the object to be matched to obtain an interaction feature of the object to be matched as a second interaction feature.

[0178] Further optionally, when the processor 602 calculates the degree of matching between the object to be queried and the at least one object to be matched based on the first interaction feature and the respective interaction features of at least one object to be matched, it is specifically used to: in a query link model, perform a feature interaction operation on the first interaction feature and the second interaction feature to obtain a third interaction feature; in a scoring layer of the query link model, score the third interaction feature according to the parameters of the scoring layer to obtain the degree of matching between the object to be queried and the object to be matched.

[0179] Further optionally, when the processor 602 performs feature interaction operations on the first interaction feature and the second interaction feature to obtain the third interaction feature, it is specifically used to: in the second interaction layer in the query link model, based on the attention mechanism, perform bidirectional interaction processing on the first interaction feature and the second interaction feature to obtain the bidirectional interaction feature vector of the object to be queried and the bidirectional interaction feature vector of the object to be matched; and fuse the bidirectional interaction feature vector of the object to be queried and the bidirectional interaction feature vector of the object to be matched to obtain the third interaction feature.

[0180] Further optionally, the first interaction feature includes multiple feature vectors corresponding to the multimodal description features of the object to be queried, and the second interaction feature includes multiple feature vectors corresponding to the multimodal representation features of the object to be matched; when the processor 602 performs bidirectional interaction processing on the first interaction feature and the second interaction feature based on the attention mechanism to obtain the bidirectional interaction feature vector of the object to be queried, it is specifically used to: for any feature vector in the first interaction feature, calculate the similarity between the feature vector and multiple feature vectors in the second interaction feature to obtain multiple bidirectional interaction weights; according to the multiple bidirectional interaction weights, perform weighted calculation on the multiple feature vectors in the second interaction feature to obtain the bidirectional interaction vector of the feature; splice the bidirectional interaction vectors of the multiple feature vectors contained in the first interaction feature to obtain the bidirectional interaction feature vector of the object to be queried.

[0181] Further, if Figure 6 As shown, the query device also includes: a communication component 603, a display 604, a power component 605, an audio component 606 and other components. Figure 6 Only some components are shown schematically, which does not mean that the query device only includes Figure 6 Components shown.

[0182] In this embodiment, when processing a query request, an interactive operation is performed based on the multimodal description features of the object to be queried to obtain interactive features. Based on the interactive features of the object to be queried and the interactive features of the object to be matched, the matching degree of the object to be queried and the object to be matched is calculated, and based on the matching degree, the target object that matches the object to be queried is determined from the objects to be matched. In this embodiment, the descriptive features of multiple modes are used, and the descriptive features of multiple modes are interactively operated, so that the descriptive features of different modes and the interactive relationships between them can work together in the process of object recognition, thereby reducing or eliminating the ambiguity of the features of a single mode, which is conducive to effectively improving the accuracy of object recognition.

[0183] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by the query device in the above method embodiment.

[0184] Figure 7 A schematic diagram of the structure of a model training device provided by an exemplary embodiment of the present application is shown. Figure 7 As shown, the model training device includes: a memory 701 and a processor 702.

[0185] The memory 701 is used to store computer programs and can be configured to store various other data to support operations on the model training device. Examples of such data include instructions for any application or method operating on the model training device, contact data, phone book data, messages, pictures, videos, etc.

[0186] The processor 702 is coupled to the memory 701 and is used to execute the computer program in the memory 701, so as to: obtain the multimodal description feature of the object to be queried according to the query description information of the object to be queried; perform feature interaction operations on the multimodal description feature of the object to be queried and the multimodal representation feature of the object to be matched, respectively, to obtain a first interaction feature and a second interaction feature; calculate the matching degree of the first interaction feature and the second interaction feature according to the model parameters of the query link model; update the model parameters according to the matching degree and the true value of the matching degree between the object to be queried and the object to be matched, so as to optimize the query link model.

[0187] Further optionally, the multimodal description features of the object to be queried may include: at least one of: semantic features of the name of the object to be queried, contextual semantic features of the name of the object to be queried, and image features of the object to be queried.

[0188] Further optionally, the multimodal representation features of the object to be matched may include at least one of: structural features of the object to be matched, image features of the object to be matched, semantic features of the name of the object to be matched, and summary features of the object to be matched.

[0189] Further optionally, the processor 702 is also used to: perform at least one of the following operations to obtain multimodal representation features of the object to be matched: generate structural features of the object to be matched based on the object to be matched and other objects that have a set relationship; calculate image features of the object to be matched based on the image of the object to be matched; generate a word vector based on the name of the object to be matched to obtain semantic features of the name of the object to be matched; generate a word vector based on the summary information of the object to be matched to obtain summary features of the object to be matched.

[0190] Further optionally, when the processor 702 generates the structural features of the object to be matched based on the object to be matched and other objects with which a set relationship exists, the processor 702 is specifically used to: calculate other objects with which a set relationship exists with the object to be matched based on a translation embedding algorithm; and use a vector consisting of the object to be matched, the set relationship and the other objects as the structural vector of the object to be matched.

[0191] Further optionally, when the processor 702 performs a feature interaction operation on the multimodal description features of the object to be queried to obtain the first interaction feature, it is specifically used to: at the first interaction layer of the query link model, based on the attention mechanism, perform self-interaction processing on the multimodal description features of the object to be queried to obtain the self-interaction feature vectors of the multimodal description features of the object to be queried; and fuse the self-interaction feature vectors of the multimodal description features of the object to be queried to obtain the first interaction feature.

[0192] Further optionally, when the processor 702 performs self-interaction processing on the multimodal description features of the object to be queried based on the attention mechanism to obtain the self-interaction feature vectors of the multimodal description features of the object to be queried, it is specifically used to: calculate the similarity between the description features and each feature in the multimodal description features of the object to be queried for the description features of any modality in the multimodal description features of the object to be queried, and obtain multiple self-interaction weights corresponding to the description features; and perform weighted calculation on the multimodal description features of the object to be queried according to the multiple self-interaction weights to obtain the self-interaction vector of the description features.

[0193] Further optionally, when the processor 702 calculates the degree of match between the first interaction feature and the second interaction feature according to the model parameters of the query link model, it is specifically used to: in the query link model, perform a feature interaction operation on the first interaction feature and the second interaction feature to obtain a third interaction feature; in the scoring layer of the query link model, score the third interaction feature according to the parameters of the scoring layer to obtain the degree of match between the first interaction feature and the second interaction feature.

[0194] Further optionally, when the processor 702 performs feature interaction operations on the first interaction feature and the second interaction feature to obtain the third interaction feature, it is specifically used to: at the second interaction layer in the query link model, based on the attention mechanism, perform bidirectional interaction processing on the first interaction feature and the second interaction feature to obtain the bidirectional interaction feature vector of the object to be queried and the bidirectional interaction feature vector of the object to be matched; and fuse the bidirectional interaction feature vector of the object to be queried and the bidirectional interaction feature vector of the object to be matched to obtain the third interaction feature.

[0195] Further optionally, the first interaction feature includes multiple feature vectors corresponding to the multimodal description features of the object to be queried, and the second interaction feature includes multiple feature vectors corresponding to the multimodal representation features of the object to be matched; accordingly, when the processor 702 performs bidirectional interaction processing on the first interaction feature and the second interaction feature based on the attention mechanism to obtain the bidirectional interaction feature vector of the object to be queried, it is specifically used to: for any feature vector in the first interaction feature, calculate the similarity between the feature vector and multiple feature vectors in the second interaction feature to obtain multiple bidirectional interaction weights; according to the multiple bidirectional interaction weights, perform weighted calculation on multiple features in the second interaction feature to obtain the bidirectional interaction vector of the feature; splice the bidirectional interaction vectors of each of the multiple features contained in the first interaction feature to obtain the bidirectional interaction feature vector of the object to be queried.

[0196] Further, if Figure 7 As shown, the model training device also includes: a communication component 703, a display 704, a power component 705, an audio component 706 and other components. Figure 7 Only some components are shown schematically, which does not mean that the model training equipment only includes Figure 7 Components shown.

[0197] In this embodiment, the interaction features are calculated based on the multimodal features of the object to be queried and the object to be matched, and the query link model is trained based on the calculated interaction features. This allows the query link model to continuously and automatically learn the interaction relationship between the multimodal features, and automatically learn the relationship between the interaction features of the object to be queried and the interaction features of the object to be matched, which is beneficial in the actual application of the model, and accurately identifies the objects that match the query request based on the multimodal features that have an interactive relationship.

[0198] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed, can implement the various steps that can be performed by the model training device in the above method embodiment.

[0199] Above Figure 6 and Figure 7 The memory in can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0200] Above Figure 6 and Figure 7 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0201] Above Figure 6 The display in the embodiment includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0202] Above Figure 6 and Figure 7The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0203] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0204] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0205] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0206] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0207] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0208] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0209] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0210] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0211] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A query method, characterized in that: include: In response to the query request, a multimodal description feature of the object to be queried is obtained, wherein the multimodal description feature includes: at least a semantic feature and an image feature obtained by performing feature extraction on the multimodal query information; Performing a feature interaction operation on the multimodal description features of the object to be queried to obtain a first interaction feature, wherein the feature interaction operation includes: performing cross calculations between features of multiple different modalities; the first interaction feature is obtained by performing self-interaction processing on the multimodal description features of the object to be queried based on an attention mechanism, and fusing the obtained self-interaction feature vectors; The matching degree between the object to be queried and the at least one object to be matched is calculated according to the first interaction feature and the interaction features of each of the at least one object to be matched; the matching degree is obtained by scoring the interaction feature obtained by fusing the bidirectional interaction feature vectors of the object to be queried and the object to be matched; the bidirectional interaction feature vectors of the object to be queried and the object to be matched are obtained by performing bidirectional interaction processing on the first interaction feature and the second interaction feature based on an attention mechanism; the second interaction feature is obtained by performing self-interaction processing on the multimodal representation features of the object to be matched based on the attention mechanism, and then fusing the obtained self-interaction feature vectors; According to the matching degree, a target object matching the object to be queried is determined from the at least one object to be matched.

2. The method according to claim 1, characterized in that In response to the query request, the multimodal description features of the object to be queried are obtained, including: In response to the query request, obtaining first query description information provided by the user; Providing the user with at least one other descriptive information that interacts with the first query descriptive information; In response to the user's selection operation from at least one other description information, acquiring the selected description information as second query description information; A description feature of a first modality and a description feature of a second modality are respectively acquired from the first query description information and the second query description information as multimodal description features of the object to be queried.

3. The method according to claim 2, characterized in that Also includes: Acquire, from the multimodal knowledge graph, a plurality of objects to be matched to which the first query description information is adapted; From the multimodal representation information corresponding to each of the multiple objects to be matched, other representation information belonging to a different modality from the first query description information is selected as the at least one other description information.

4. The method according to claim 2, characterized in that: The first query description information includes: text description information; the second query description information includes: image description information.

5. The method according to claim 1, characterized in that Performing a feature interaction operation on the multimodal description feature of the object to be queried to obtain a first interaction feature includes: In the first interaction layer of the query link model, based on the attention mechanism, the multimodal description features of the object to be queried are subjected to self-interaction processing to obtain the self-interaction feature vectors of the multimodal description features of the object to be queried; The self-interaction feature vectors of the multimodal description features of the object to be queried are fused to obtain the first interaction feature.

6. The method according to claim 5, characterized in that Based on the attention mechanism, the multimodal description features of the object to be queried are subjected to self-interaction processing to obtain the self-interaction feature vectors of the multimodal description features of the object to be queried, including: For a description feature of any mode in the multimodal description features of the object to be queried, calculating the similarity between the description feature and each feature in the multimodal description features of the object to be queried, and obtaining a plurality of self-interaction weights corresponding to the description feature; According to the multiple self-interaction weights, weighted calculation is performed on the multimodal description features of the object to be queried to obtain a self-interaction vector of the description features.

7. The method according to claim 1, characterized in that Also includes: For any one of the at least one object to be matched, a feature interaction operation is performed on the multimodal representation feature of the object to be matched to obtain an interaction feature of the object to be matched as a second interaction feature.

8. The method according to claim 7, characterized in that Calculating the matching degree between the object to be queried and the at least one object to be matched according to the first interaction feature and the interaction features of the at least one object to be matched, including: In the query link model, a feature interaction operation is performed on the first interaction feature and the second interaction feature to obtain a third interaction feature; In the scoring layer of the query link model, the third interaction feature is scored according to the parameters of the scoring layer to obtain the matching degree between the object to be queried and the object to be matched.

9. The method according to claim 8, characterized in that Performing a feature interaction operation on the first interaction feature and the second interaction feature to obtain a third interaction feature includes: In the second interaction layer of the query link model, based on the attention mechanism, bidirectional interaction processing is performed on the first interaction feature and the second interaction feature to obtain a bidirectional interaction feature vector of the object to be queried and a bidirectional interaction feature vector of the object to be matched; The bidirectional interaction feature vector of the object to be queried and the bidirectional interaction feature vector of the object to be matched are merged to obtain the third interaction feature.

10. The method according to claim 9, characterized in that The first interaction feature includes a plurality of feature vectors corresponding to the multimodal description feature of the object to be queried, and the second interaction feature includes a plurality of feature vectors corresponding to the multimodal representation feature of the object to be matched; Based on the attention mechanism, performing bidirectional interaction processing on the first interaction feature and the second interaction feature to obtain a bidirectional interaction feature vector of the object to be queried includes: For any feature vector in the first interaction feature, calculate the similarity between the feature vector and multiple feature vectors in the second interaction feature to obtain multiple bidirectional interaction weights; performing weighted calculation on a plurality of feature vectors in the second interaction feature according to the plurality of bidirectional interaction weights to obtain a bidirectional interaction vector of the feature; The bidirectional interaction vectors of the plurality of feature vectors included in the first interaction feature are concatenated to obtain the bidirectional interaction feature vector of the object to be queried.

11. A model training method, characterized in that: include: Acquire multimodal description features of the object to be queried according to the query description information of the object to be queried, wherein the multimodal description features include: at least semantic features and image features obtained by performing feature extraction on the multimodal query information; Performing feature interaction operations on the multimodal description features of the object to be queried and the multimodal representation features of the object to be matched respectively to obtain a first interaction feature and a second interaction feature, wherein the feature interaction operation includes: performing cross calculations between features of multiple different modalities; the first interaction feature is obtained by performing self-interaction processing on the multimodal description features of the object to be queried based on an attention mechanism, and then fusing the self-interaction feature vector obtained; calculating the matching degree of the first interaction feature and the second interaction feature according to the model parameters of the query link model; the matching degree is obtained by scoring the interaction feature obtained by fusing the bidirectional interaction feature vectors of the object to be queried and the object to be matched; the bidirectional interaction feature vectors of the object to be queried and the object to be matched are obtained by performing bidirectional interaction processing on the first interaction feature and the second interaction feature based on an attention mechanism; the second interaction feature is obtained by performing self-interaction processing on the multimodal representation features of the object to be matched based on an attention mechanism, and then fusing the self-interaction feature vector obtained; The model parameters are updated according to the matching degree and the true value of the matching degree between the object to be queried and the object to be matched, so as to optimize the query link model.

12. The method according to claim 11, characterized in that The multimodal description features of the object to be queried include: at least one of: semantic features of the name of the object to be queried, contextual semantic features of the name of the object to be queried, and image features of the object to be queried.

13. The method according to claim 11, characterized in that Also includes: Perform at least one of the following operations to obtain the multimodal representation features of the object to be matched: Generating structural features of the object to be matched according to the object to be matched and other objects having a set relationship; Calculating image features of the object to be matched according to the image of the object to be matched; Generate a word vector according to the name of the object to be matched, and obtain the semantic features of the name of the object to be matched; A word vector is generated according to the summary information of the object to be matched, and a summary feature of the object to be matched is obtained.

14. The method according to claim 13, characterized in that Generating structural features of the object to be matched according to the object to be matched and other objects having a set relationship, including: Based on the translation embedding algorithm, calculating other objects that have a set relationship with the object to be matched; A vector consisting of the object to be matched, the set relationship and the other objects is used as a structure vector of the object to be matched.

15. The method according to claim 11, characterized in that Performing a feature interaction operation on the multimodal description feature of the object to be queried to obtain a first interaction feature includes: In the first interaction layer of the query link model, based on the attention mechanism, the multimodal description features of the object to be queried are subjected to self-interaction processing to obtain the self-interaction feature vectors of the multimodal description features of the object to be queried; The self-interaction feature vectors of the multimodal description features of the object to be queried are fused to obtain the first interaction feature.

16. The method according to claim 11, characterized in that Calculating the matching degree between the first interaction feature and the second interaction feature according to the model parameters of the query link model includes: In the query link model, a feature interaction operation is performed on the first interaction feature and the second interaction feature to obtain a third interaction feature; In the scoring layer of the query link model, the third interaction feature is scored according to the parameters of the scoring layer to obtain the matching degree between the first interaction feature and the second interaction feature.

17. A query device, characterized in that: include: A first feature acquisition module is used to: respond to a query request and acquire multimodal description features of the object to be queried, wherein the multimodal description features include: at least semantic features and image features obtained by performing feature extraction on multimodal query information; A first interaction module is used to: perform a feature interaction operation on the multimodal description features of the object to be queried to obtain a first interaction feature, wherein the feature interaction operation includes: performing a cross calculation between features of multiple different modalities; the first interaction feature is obtained by performing self-interaction processing on the multimodal description features of the object to be queried based on an attention mechanism, and then fusing the obtained self-interaction feature vector; A matching degree calculation module is used to: calculate the matching degree of the object to be queried and the at least one object to be matched according to the first interaction feature and the interaction features of each of the at least one object to be matched; the matching degree is obtained by scoring the interaction feature obtained by fusing the bidirectional interaction feature vectors of the object to be queried and the object to be matched; the bidirectional interaction feature vectors of the object to be queried and the object to be matched are obtained by performing bidirectional interaction processing on the first interaction feature and the second interaction feature based on an attention mechanism; the second interaction feature is obtained by performing self-interaction processing on the multimodal representation features of the object to be matched based on an attention mechanism, and then fusing the obtained self-interaction feature vectors; The target object determination module is used to determine, according to the matching degree, a target object that matches the object to be queried from the at least one object to be matched.

18. A model training device, characterized in that: include: A first feature acquisition module is used to: acquire multimodal description features of the object to be queried according to the query description information of the object to be queried, wherein the multimodal description features include: at least semantic features and image features obtained by performing feature extraction on the multimodal query information; A first interaction module is used to: perform feature interaction operations on the multimodal description features of the object to be queried and the multimodal representation features of the object to be matched, respectively, to obtain a first interaction feature and a second interaction feature, wherein the feature interaction operation includes: cross calculation between features of multiple different modalities; the first interaction feature is obtained by performing self-interaction processing on the multimodal description features of the object to be queried based on an attention mechanism, and then fusing the obtained self-interaction feature vectors; A matching degree calculation module is used to: calculate the matching degree of the first interaction feature and the second interaction feature according to the model parameters of the query link model; the matching degree is obtained by scoring the interaction feature obtained by fusing the bidirectional interaction feature vectors of the object to be queried and the object to be matched; the bidirectional interaction feature vectors of the object to be queried and the object to be matched are obtained by performing bidirectional interaction processing on the first interaction feature and the second interaction feature based on an attention mechanism; the second interaction feature is obtained by performing self-interaction processing on the multimodal representation features of the object to be matched based on the attention mechanism, and then fusing the obtained self-interaction feature vectors; The parameter optimization module is used to update the model parameters according to the matching degree and the true value of the matching degree between the object to be queried and the object to be matched, so as to optimize the query link model.

19. An electronic device, characterized in that: include: A memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the steps in the method described in any one of claims 1-16.

20. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the steps in the method according to any one of claims 1 to 16 can be implemented.

Images