Answer information acquisition method, device, equipment and storage medium

Abstract

The present application discloses a method, device, equipment and storage medium for obtaining answer information, and belongs to the field of computer technology. The method includes: based on a target image, obtaining description information of the target image, the description information includes multiple descriptor information; based on the question information of the target image, determining the correlation between each descriptor information in the description information and the question information; based on the determined correlation, weakening at least one descriptor information in the description information to obtain the processed description information; based on the question information and the processed description information, determining the answer information of the question information. This scheme can weaken the noise information in the target image and reduce the interference of the noise information, so as to more accurately determine the answer information of the question information and improve the accuracy of visual question answering.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for recommending answer information. Background Art

[0002] Visual Question Answering (VQA) is a typical task that combines the two fields of computer vision and natural language processing, and has received widespread attention. The VQA system can generate an answer based on a specified image and the question asked about the specified image. In short, VQA is to ask and answer questions about specified images. For example, the specified image is an image of a football game, and the question asked about the specified image is "How many players are there in the photo?" The VQA system can take the specified image and question information as input, combine these two parts of information, and generate an answer (for example, 11) as output.

[0003] In the related art, the VQA system processes the input image to obtain the description information of the image, and based on the description information of the image and the input question information, determines the answer information that matches the image and the question information from multiple candidate answer information. However, it is found in actual applications that the accuracy of the VQA system is low. Summary of the invention

[0004] The embodiment of the present application provides a method, device, equipment and storage medium for obtaining answer information, which improves the accuracy of visual question answering. The technical solution is as follows:

[0005] On the one hand, a method for obtaining answer information is provided, the method comprising: based on a target image, obtaining description information of the target image, the description information comprising multiple descriptor information; based on question information of the target image, determining the correlation between each descriptor information in the description information and the question information; based on the determined correlation, weakening at least one descriptor information in the description information to obtain processed description information; based on the question information and the processed description information, determining answer information of the question information.

[0006] On the one hand, a device for acquiring answer information is provided, the device comprising: an information acquisition module, for acquiring description information of a target image based on a target image, the description information comprising a plurality of descriptor information; a correlation determination module, for determining the correlation between each descriptor information in the description information and the question information based on question information of the target image; a processing module, for weakening at least one descriptor information in the description information based on the determined correlation to obtain processed description information; and an answer determination module, for determining answer information of the question information based on the question information and the processed description information.

[0007] In one possible implementation, the description information includes relationship description information, the relationship description information includes multiple relationship description sub-information, and different relationship description sub-information is used to describe different relationships between objects in the target image; the question information is used to represent the problem associated with the object in the target image; the correlation determination module is used to determine the correlation between each relationship description sub-information in the relationship description information and the question information based on the question information, and the correlation indicates the degree of correlation between the relationship represented by the relationship description sub-information and the question information.

[0008] In a possible implementation, the relationship descriptor information refers to a relationship graph of the target image, and different relationship descriptor information refers to different types of relationship graphs of the target image; a node in the relationship graph refers to an object in the target image, and the node feature of the node refers to the object feature of the object; the edge between two nodes in the relationship graph refers to the relationship between the corresponding two objects, the edge feature of the edge refers to the relationship feature of the relationship between the corresponding two objects, and the direction of the edge is the direction indicated by the relationship between the two objects; or, the edge between two nodes in the relationship graph refers to the correlation under the corresponding relationship between the two objects, and the direction of the edge is the direction indicated by the correlation between the corresponding two objects; the correlation determination module is used to determine the first global feature of each relationship graph of the multiple relationship graphs of the target image; determine the second global feature of the question information; and determine the similarity between the first global feature of each relationship graph and the second global feature as the correlation between each relationship graph and the question information.

[0009] In one possible implementation, the processing module includes: a weight determination unit, used to determine the weight of each relationship graph based on the determined association degree; a fusion unit, used to, when the processed description information includes the third global feature of the target image, perform weighted fusion of the first global features of the multiple relationship graphs based on the weight of each relationship graph, to obtain the third global feature of the target image; or, when the processed description information includes the target relationship graph of the target image, perform weighted fusion of the multiple relationship graphs based on the weight of each relationship graph, to obtain the target relationship graph of the target image.

[0010] In one possible implementation, the description information includes object description information, the object description information includes multiple object description sub-information, different object description sub-information is used to describe different objects in the target image, and the question information is used to represent the problem associated with the object in the target image; the correlation determination module is used to determine the correlation between each object description sub-information in the object description information and the question information based on the question information, and the correlation is used to represent the degree of correlation between the object represented by the object description sub-information and the question information.

[0011] In one possible implementation, the correlation determination module includes: a fusion unit, used to fuse each object description sub-information in the description information with the question information to obtain multiple fused object description sub-information; a determination unit, used to determine the correlation between every two fused object description sub-information based on the multiple fused object description sub-information; the determination unit is also used to determine the correlation between each fused object description sub-information and the question information based on the correlation and value of other fused object description sub-information in the multiple fused object description sub-information except the fused object description sub-information and the fused object description sub-information.

[0012] In one possible implementation, the object description sub-information refers to an object feature; the fusion unit is used to obtain a text feature of each word in the question information; for each object feature in the description information, the similarity between the object feature and each text feature is obtained, and based on the obtained similarity, each text feature is fused with the object feature to obtain a fused object feature.

[0013] In one possible implementation, the object descriptor information refers to an object feature; the determination unit is used to obtain, based on each fused object feature, a query feature, a key feature, and a value feature of the fused object feature, obtain a similarity between the query feature of the fused object feature and a key feature of other fused object features among multiple fused object features except the fused object feature, and determine, based on the obtained similarity, a degree of association between the fused object feature and the other fused object features.

[0014] In a possible implementation, the processing module is used to weaken at least one fused object description sub-information in the description information based on the determined correlation degree to obtain the processed description information.

[0015] In one possible implementation, the processing module is used to determine the weight of each descriptor information based on the determined degree of association, and weaken the multiple descriptor information based on the weight of each descriptor information to obtain the processed description information; or, the processing module is used to determine at least one descriptor information from the description information based on the determined degree of association, and weaken the at least one descriptor information to obtain the processed description information.

[0016] In one possible implementation, the processing module is used to determine the weight of each descriptor information in the at least one descriptor information based on the determined correlation, and based on the weight of each descriptor information, weaken the at least one descriptor information in the description information to obtain the processed description information; or, the processing module is used to delete the at least one descriptor information from the description information to obtain the processed description information.

[0017] On the one hand, a computer device is provided, which includes one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the operations performed by the answer information acquisition method in any possible implementation method as described above.

[0018] On the one hand, a computer-readable storage medium is provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the operations performed by the answer information acquisition method in any possible implementation manner as described above.

[0019] On the one hand, a computer program or a computer program product is provided, wherein the computer program or the computer program product comprises: a computer program code, wherein when the computer program code is executed by a computer, the computer implements the operations performed by the answer information acquisition method in any possible implementation manner as described above.

[0020] The answer information acquisition method, device, equipment and storage medium provided in the embodiments of the present application take into account that there is a lot of information in the target image that has nothing to do with the question information, and this information that is irrelevant to the question information can be regarded as noise information. Therefore, by determining the correlation between each descriptor information of the target image and the question information, the present application can find the descriptor information with a low correlation with the question information. Based on the determined correlation, at least one descriptor information in the description information is weakened, which can weaken the descriptor information with a low correlation with the question information, thereby achieving the effect of weakening the noise information in the target image and reducing the interference of the noise information, so that the answer information of the question information can be determined more accurately, thereby improving the accuracy of visual question answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0023] Figure 2 is a flow chart of a method for obtaining answer information provided in an embodiment of the present application;

[0024] Figure 3 is a flow chart of a method for obtaining answer information provided in an embodiment of the present application;

[0025] Figure 4 is a flow chart of a weakening processing method provided in an embodiment of the present application;

[0026] Figure 5 is a flow chart of a method for obtaining answer information provided in an embodiment of the present application;

[0027] Figure 6 is a flow chart of a weakening processing method provided in an embodiment of the present application;

[0028] Figure 7 is a flow chart of a method for obtaining answer information provided in an embodiment of the present application;

[0029] Figure 8 It is a schematic diagram of the structure of an answer information acquisition device provided in an embodiment of the present application;

[0030] Fig. 9 is a schematic diagram of the structure of another answer information acquisition device provided in an embodiment of the present application;

[0031] Fig.10 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0032] Fig.11 It is a schematic diagram of the structure of the server provided in the embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0034] It is understood that the terms "first", "second", etc. used in this application can be used in this article to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, a first object can be referred to as a second object, and similarly, a second object can be referred to as a first object without departing from the scope of this application.

[0035] The terms "at least one", "multiple", "each" and "any" used in this application, at least one includes one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple. For example, multiple information includes 3 information, and each refers to each of the 3 information, and any refers to any one of the 3 information, which can be the first, the second or the third.

[0036] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0037] The answer information acquisition method provided in the embodiment of the present application is executed by a computer device. In one possible implementation, the computer device is a terminal, for example, the terminal is any type of terminal such as a desktop computer, a tablet computer or a mobile phone. In another possible implementation, the computer device is a server. For example, the server can be a server, or a server cluster composed of several servers, or a cloud computing service center. In another possible implementation, the computer device includes a terminal and a server.

[0038] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application, such as Figure 1 As shown, the implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network.

[0039] Optionally, the terminal 101 is any type of terminal such as a desktop computer, a tablet computer or a mobile phone, etc. The server 102 is a server, or a server cluster composed of several servers, or a cloud computing service center.

[0040] The target application provided by the server 102 is installed on the terminal 101, and the terminal 101 can realize functions such as data transmission and message interaction through the target application. Optionally, the target application is an application in the operating system of the terminal 101, or an application provided by a third party. For example, the target application is a question-and-answer application, which has the function of providing answers. Of course, the question-and-answer application can also have other functions, such as commenting function, sharing function, etc.

[0041] Optionally, the terminal 101 sends the target image and the question information input by the user for the target image to the server 102. The server 102 determines the answer information of the question information based on the target image and the question information, and sends the answer information to the terminal 101. The terminal 101 receives the answer information and displays the answer information.

[0042] The visual question answering method provided in the embodiments of the present application can be applied to any question answering scenario.

[0043] For example, in the application of e-commerce question and answer scenarios.

[0044] The user can input a picture of the target item and question information about the picture (for example, how much is the target item) in the e-commerce application. If the answer information acquisition method provided in the embodiment of the present application is adopted, the e-commerce application can more accurately obtain the answer information of the question information.

[0045] It should be noted that the embodiments of the present application are only illustrative of e-commerce question and answer scenarios and do not limit the question and answer scenarios.

[0046] Figure 2 This is a flowchart of a method for obtaining answer information provided by an embodiment of the present application. This embodiment of the present application is illustrated by taking a computer device as an example of an execution subject. The embodiment includes:

[0047] 201. A computer device obtains description information of a target image based on a target image, where the description information includes multiple description sub-information.

[0048] The target image may be any image. For example, the target image may be a local image of the computer device; for another example, the target image may be an image captured by the computer device; for another example, the target image may be an image acquired by the computer device from other devices.

[0049] The target image may be an image of at least one object, that is, the target image may include at least one object. The object may be any object such as a person, an object, a tree, a flower, a beach, a mountain, etc. The embodiment of the present application does not limit the content of the target image.

[0050] The description information of the target image is information that describes the content of the target image. For example, the description information is information used to describe an object in the target image. The description information includes multiple descriptor information, and the multiple descriptor information is used to describe the target image from different angles. For example, the multiple descriptor information is used to describe different objects in the target image (the target image is a photo of the seaside, descriptor information 1 is used to describe the beach in the target image, descriptor information 2 is used to describe the ocean in the target image, and descriptor information 3 is used to describe the coconut trees in the target image); for another example, the multiple descriptor information is used to describe different attribute information of the same object in the target image (the target image is a photo of the beach, descriptor information 1 is used to describe the color of the beach, and descriptor information 2 is used to describe the size of the beach).

[0051] Optionally, the target image includes multiple objects, and the description information is used to describe the multiple objects. The description information may describe each of the multiple objects separately, or may describe at least two of the multiple objects together. In an embodiment of the present application, the description information describing at least two of the multiple objects together may be: the description information describes the relationship between at least two of the multiple objects.

[0052] In a possible implementation, the description information includes relationship description information, and the relationship description information is used to describe the relationship between objects in the target image. The relationship description information includes multiple relationship description sub-information; the computer device obtains the description information of the target image based on the target image, including: the computer device obtains the multiple relationship description sub-information of the target image based on the target image.

[0053] Optionally, different relationship description sub-information is used to describe the relationship between different objects in the target image. For example, the target image includes object 1, object 2, and object 3, and the multiple relationship description sub-information includes 3 relationship description sub-information: relationship description sub-information 1, relationship description sub-information 2, and relationship description sub-information 3.

[0054] Among them, relationship description sub-information 1 is used to describe the relationship between object 1 and object 2, relationship description sub-information 2 is used to describe the relationship between object 2 and object 3, and relationship description sub-information 3 is used to describe the relationship between object 1 and object 3.

[0055] Optionally, different relationship descriptor information is used to describe different relationships between objects in the target image. In an embodiment of the present application, different relationship descriptor information is used to describe different relationships between objects in the target image, which means that different relationship descriptor information is used to describe different types of relationships between objects in the target image. The computer device obtains multiple relationship descriptor information of the target image based on the target image, including at least one of the following: (1) The relationship description information includes semantic relationship descriptor information, which represents the semantic relationship between objects in the target image; the computer device obtains the semantic relationship descriptor information of the target image based on the target image. (2) The relationship description information includes spatial relationship descriptor information, which represents the spatial relationship between objects in the target image; the computer device obtains the spatial relationship descriptor information of the target image based on the target image. (3) The relationship description information includes implicit relationship descriptor information, which represents the implicit relationship between objects in the target image; the computer device obtains the spatial implicit descriptor information of the target image based on the target image.

[0056] In another possible implementation, the description information includes object description information, and the object description information includes multiple pieces of object description sub-information. Optionally, different object description sub-information is used to describe different objects in the target image; optionally, different object description sub-information is used to describe different attributes of the same object in the target image. The computer device acquires the description information of the target image based on the target image, including: the computer device acquires the multiple pieces of object description sub-information of the target image based on the target image.

[0057] In another possible implementation, the description information includes relationship description information and object description information. The computer device acquires the description information of the target image based on the target image, including: the computer device acquires the relationship description information and object description information of the target image based on the target image.

[0058] 202. The computer device determines, based on the question information of the target image, a degree of association between each descriptor sub-information in the description information and the question information.

[0059] The question information is the question information raised for the target image, and the question information may be input by the user or obtained from the network. The embodiment of the present application does not limit the question information. For example, the target image is a picture of a football game, and the question information of the target image is "how many players are there in the picture", "what sport are the people in the picture doing", etc.

[0060] In general, the target image includes information related to the question information as well as information unrelated to the question information. For example, the target image is a picture of a football game, and the question information of the target image is "how many players are there in the picture". In this case, the information about the location of the audience seats in the target image is irrelevant to the question information, but the information about the players wearing uniforms in the target image is relevant to the question information.

[0061] 203. The computer device weakens at least one descriptor information in the description information based on the determined association degree to obtain processed description information.

[0062] In the embodiment of the present application, at least one descriptor information may be part of the description information or all of the description information. Weakening the descriptor information means: reducing the influence of the descriptor information in the description information.

[0063] In a possible implementation, the at least one descriptor information is all the information in the description information. The computer device weakens the at least one descriptor information in the description information based on the determined association degree to obtain the processed description information, including: determining the weight of each descriptor information based on the determined association degree, and weakening the multiple descriptor information based on the weight of each descriptor information to obtain the processed description information.

[0064] When the computer device determines the weight of each descriptor information based on the determined relevance, the higher the relevance of the descriptor information, the higher the weight; the lower the relevance of the descriptor information, the lower the weight. In this way, the higher the relevance of the descriptor information to the question information, the higher the proportion of the descriptor information in the description information, and the more accurately the computer device can determine the answer information based on the descriptor information.

[0065] In another possible implementation, the at least one descriptor information is part of the description information. The computer device weakens the at least one descriptor information in the description information based on the determined association degree to obtain the processed description information, including: determining at least one descriptor information from the description information based on the determined association degree, weakening the at least one descriptor information to obtain the processed description information.

[0066] Optionally, the computer device weakens the at least one descriptor information to obtain the processed description information, including: determining the weight of each descriptor information in the at least one descriptor information based on the determined correlation degree, and weakening the at least one descriptor information in the description information based on the weight of each descriptor information to obtain the processed description information.

[0067] Optionally, the computer device performs weakening processing on the at least one descriptor information to obtain the processed description information, including: deleting the at least one descriptor information from the description information to obtain the processed description information.

[0068] In another possible implementation, the computer device weakens at least one descriptive sub-information in the description information based on the determined correlation to obtain the processed description information, including: weakening the descriptive sub-information in the description information whose correlation with the problem information is less than a correlation threshold to obtain the processed description information.

[0069] Among them, the correlation threshold is any value, which can be an empirical value, a value set by a technician, or a default value of a computer device. The embodiment of the present application does not limit the correlation threshold.

[0070] In another possible implementation, a computer device weakens at least one descriptor information in the description information based on the determined degree of association to obtain the processed description information, including: determining a first number of degrees of association from the determined degree of association, the first number of degrees of association being smaller than other degrees of association in the determined degree of association except the first number of degrees of association; weakening the descriptor information corresponding to the first number of degrees of association to obtain the processed description information.

[0071] For example, after obtaining the association degree, the computer device sorts the obtained association degree in order of size. If the association degree is sorted in order from large to small, the last first number of association degrees are selected, and the descriptor information corresponding to the last first number of association degrees is weakened to obtain the processed description information. If the association degree is sorted in order from small to large, the first first number of association degrees are selected, and the descriptor information corresponding to the first first number of association degrees is weakened to obtain the processed description information.

[0072] 204. The computer device determines answer information of the question information based on the question information and the processed description information.

[0073] In the embodiment of the present application, answer information matching the question information can be determined from multiple candidate answers based on the question information and the processed description information. The multiple candidate answers can be answers in the answer library or answer information recalled based on the question information and the target image. The embodiment of the present application does not limit the multiple candidate answers.

[0074] The answer information acquisition method provided in the embodiment of the present application takes into account that there is a lot of information in the target image that has nothing to do with the question information, and this information that is irrelevant to the question information can be regarded as noise information. Therefore, by determining the correlation between each descriptor information of the target image and the question information, the present application can find the descriptor information with a low correlation with the question information. Based on the determined correlation, at least one descriptor information in the description information is weakened, which can weaken the descriptor information with a low correlation with the question information, thereby achieving the effect of weakening the noise information in the target image and reducing the interference of the noise information, so that the answer information of the question information can be determined more accurately, thereby improving the accuracy of visual question answering.

[0075] It should be noted that, in the embodiments of the present application, the description information may include relationship description information, which includes multiple relationship description sub-information; or, the description information includes object description information, which includes multiple object description sub-information; or, the description information includes relationship description information and object description information. Figure 3 The embodiment shown in the figure is used as an example to illustrate the situation where "the description information includes relationship description information, and the relationship description information includes multiple relationship description sub-information". Figure 5 The embodiment shown in the figure is used as an example to illustrate the situation where "the description information includes object description information, and the object description information includes multiple object description sub-information". Figure 6 The illustrated embodiment is taken as an example to exemplarily illustrate the situation where “the description information includes object description information and relationship description information”.

[0076] Figure 3 This is a flowchart of a method for obtaining answer information provided by an embodiment of the present application. This embodiment of the present application is illustrated by taking a computer device as an example of an execution subject. The embodiment includes:

[0077] 301. A computer device obtains description information of a target image based on a target image, where the description information includes relationship description information, and the relationship description information includes a plurality of relationship description sub-information.

[0078] Optionally, different relationship descriptor information is used to describe different relationships between objects in the target image. For example, the description information includes semantic relationship descriptor information, spatial relationship descriptor information, and implicit relationship descriptor information. Among them, the semantic relationship descriptor information is used to describe the semantic relationship between objects in the target image; the spatial relationship descriptor information is used to describe the spatial relationship between objects in the target image; and the implicit relationship descriptor information is used to describe the implicit relationship between objects in the target image.

[0079] Optionally, different relationship descriptor information is used to describe the relationship between different objects in the target image. For example, the description information includes relationship descriptor information 1, relationship descriptor information 2, and relationship descriptor information 3. Relationship descriptor information 1 is used to describe the relationship between object 1 and object 2; relationship descriptor information 2 is used to describe the relationship between object 2 and object 3; relationship descriptor information 3 is used to describe the relationship between object 1 and object 3.

[0080] 302. The computer device determines, based on the question information of the target image, a degree of association between each piece of relationship description sub-information in the description information and the question information.

[0081] The question information is used to indicate a question associated with the content of the target image. For example, if the target image is a picture of a football game, the question information is “how many players are there in the picture”. Optionally, the target image includes at least one object, and the question information is used to indicate a question associated with the object in the target image.

[0082] Since the question information represents a question related to the content of the target image, the higher the similarity between the question information and a certain content in the target image, the higher the relevance of the question information to the content. For example, the target image is a picture of a football game, and the question information is "how many athletes are there in the picture", and the question information contains "athletes", so the content in the target image that is more similar to the question information is the content related to "athletes", and the content in the target image that is related to "athletes" is also the content that is more related to the question information.

[0083] In one possible implementation, the computer device determines the relevance between each piece of relationship description sub-information in the description information and the question information based on the question information of the target image, including: the computer device determines the similarity between each piece of relationship description sub-information in the description information and the question information based on the question information of the target image; and determines the relevance between each piece of relationship description sub-information and the question information based on the similarity between each piece of relationship description sub-information and the question information.

[0084] Optionally, the computer device determines the relevance between each piece of relationship description sub-information and the question information based on the similarity between each piece of relationship description sub-information and the question information, including: the computer device determines the relevance between each piece of relationship description sub-information and the question information as the relevance between the relationship description sub-information and the question information. Optionally, the computer device determines the relevance between each piece of relationship description sub-information and the question information based on the similarity between each piece of relationship description sub-information and the question information, including: normalizing the similarity between each piece of relationship description sub-information and the question information to obtain the relevance between each piece of relationship description sub-information and the question information.

[0085] It should be noted that the description information, relationship description sub-information and other information in the embodiments of the present application can be in any form of information, and the embodiments of the present application do not limit the description information and relationship description sub-information. For example, the relationship description sub-information refers to a relationship label, for example, the relationship label is "object A squatting on the right side of object B", "object A and object B are looking at each other", etc. For another example, the relationship description sub-information refers to a relationship graph of the target image, for example, a semantic relationship graph, a spatial relationship graph, an implicit relationship graph, etc. of the target image.

[0086] In one possible implementation, the relationship descriptor information refers to a relationship graph of the target image, and different relationship descriptor information refers to different types of relationship graphs of the target image; a node in the relationship graph refers to an object in the target image, and the node feature of the node refers to the object feature of the object.

[0087] In addition, the edge between two nodes in the relationship graph refers to the relationship between the corresponding two objects, the edge feature of the edge refers to the relationship feature of the relationship between the corresponding two objects, and the direction of the edge is the direction indicated by the relationship between the corresponding two objects; or, the edge between two nodes in the relationship graph refers to the degree of association under the corresponding relationship between the corresponding two objects, and the direction of the edge is the direction indicated by the degree of association between the corresponding two objects. The embodiment of the present application does not limit the form of expression of the relationship graph.

[0088] The computer device determines the degree of association between each relationship description sub-information in the relationship description information and the question information based on the question information, including: the computer device determines a first global feature of each relationship graph among multiple relationship graphs of the target image; determines a second global feature of the question information; and determines the similarity between the first global feature and the second global feature of each relationship graph as the degree of association between each relationship graph and the question information.

[0089] It should be noted that any method of extracting global features can be used to obtain the first global feature and the second global feature in the embodiments of the present application, and the embodiments of the present application will not be described one by one here.

[0090] 303. The computer device weakens at least one piece of relationship description sub-information in the description information based on the determined correlation degree to obtain processed description information.

[0091] It should be noted that the above step 303 is similar to the above step 203. Any weakening processing method provided in the above step 203 is also applicable to step 303, which will not be described in detail here.

[0092] It should be noted that the embodiment of the present application only takes the example of the relationship descriptor information referring to a relationship graph, and exemplifies "based on the weight of each descriptor information, weakening at least one descriptor information in the description information". In one possible implementation, the processed description information includes the third global feature of the target image, and the computer device weakens at least one descriptor information in the description information based on the determined degree of association to obtain the processed description information, including: determining the weight of each relationship graph based on the determined degree of association; based on the weight of each relationship graph, weightedly fusing the first global features of the multiple relationship graphs to obtain the third global feature of the target image.

[0093] In another possible implementation, the processed description information includes a target relationship graph of the target image. The computer device weakens at least one descriptor information in the description information based on the determined degree of association to obtain the processed description information, including: based on the weight of each relationship graph, weighted fusion of multiple relationship graphs to obtain the target relationship graph of the target image.

[0094] For example, Figure 4 As shown, the correlation between the semantic relationship graph, spatial relationship graph and implicit relationship graph of the target image and the question information is determined respectively, and according to the corresponding correlations of the semantic relationship graph, spatial relationship graph and implicit relationship graph, the semantic relationship graph, spatial relationship graph and implicit relationship graph are fused to obtain the target relationship graph of the target image.

[0095] 304. The computer device determines answer information of the question information based on the question information and the processed description information.

[0096] The step 304 is the same as the above step 204 and will not be described in detail here.

[0097] The answer information acquisition method provided in the embodiment of the present application takes into account that there is a lot of information in the target image that has nothing to do with the question information, and this information that is irrelevant to the question information can be regarded as noise information. Therefore, the present application can find the relationship descriptor information with a low correlation with the question information by determining the correlation between each relationship descriptor information of the target image and the question information. Based on the determined correlation, at least one relationship descriptor information in the description information is weakened, which can weaken the relationship descriptor information with a low correlation with the question information, thereby achieving the effect of weakening the noise information in the target image and reducing the interference of the noise information, so that the answer information of the question information can be determined more accurately, thereby improving the accuracy of visual question answering.

[0098] Figure 5 This is a flowchart of a method for obtaining answer information provided by an embodiment of the present application. This embodiment of the present application is illustrated by taking a computer device as an example of an execution subject. The embodiment includes:

[0099] 501. A computer device obtains description information of a target image based on a target image, where the description information includes object description information, and the object description information includes a plurality of object description sub-information.

[0100] In an embodiment of the present application, the target image includes at least one object, which may be any object such as a person, an object, a flower, a tree, a beach, a mountain, etc. The embodiment of the present application does not limit the objects included in the target image. Optionally, different object description sub-information is used to describe different objects in the target image. For example, object description sub-information 1 is used to describe person A in the target image, and object description sub-information 2 is used to describe person B in the target image. Optionally, different object description sub-information is used to describe different attributes of objects in the target image. For example, object description sub-information 1 is used to describe the identity of the object, object description sub-information 2 is used to describe the location of the object, object description sub-information 3 is used to describe the appearance of the object, object description sub-information 4 is used to describe the behavior of the object, etc.

[0101] 502. The computer device determines the relevance between each object description sub-information in the description information and the question information based on the question information of the target image.

[0102] In the embodiment of the present application, the target image includes at least one object, and the question information is used to represent a question associated with the object in the target image. For example, the target image is a picture of a football game, and the question information is "how many players are there in the picture?"

[0103] Since the question information is a question associated with the object in the target image, the higher the similarity between the question information and the object descriptor information of the target image, the higher the correlation between the question information and the object descriptor information, and a more accurate answer can be obtained based on the object descriptor information.

[0104] In one possible implementation, the computer device determines the correlation between each object description sub-information in the description information and the question information based on the question information, including: the computer device determines the similarity between each object description sub-information in the description information and the question information based on the question information; and determines the correlation between each object description sub-information and the question information based on the similarity between each object description sub-information and the question information.

[0105] Optionally, the computer device determines the degree of association between each object description sub-information and the question information based on the similarity between each object description sub-information and the question information, including: determining the similarity between each object description sub-information and the question information as the degree of association between each object description sub-information and the question information. Optionally, the computer device determines the degree of association between each object description sub-information and the question information based on the similarity between each object description sub-information and the question information, including: normalizing the similarity between each object description sub-information and the question information to obtain the degree of association between each object description sub-information and the question information.

[0106] In some embodiments, the question information is associated with multiple objects in the target image, and for this purpose, the embodiments of the present application also provide another method for determining the degree of association. In one possible implementation, the computer device determines the degree of association between each object description sub-information in the description information and the question information based on the question information, including: fusing each object description sub-information in the description information with the question information to obtain multiple fused object description sub-information; based on the multiple fused object description sub-information, determining the degree of association between each two fused object description sub-information; for each fused object description sub-information, determining the degree of association between the fused object description sub-information and the question information based on the degree of association and value of other fused object description sub-information in the multiple fused object description sub-information except the fused object description sub-information and the fused object description sub-information.

[0107] Optionally, the object description sub-information refers to an object feature; the computer device fuses each object description sub-information in the description information with the question information to obtain multiple fused object description sub-information, including: the computer device obtains the text feature of each word in the question information; for each object feature in the description information, obtains the similarity between the object feature and each text feature, and based on the obtained similarity, fuses each text feature with the object feature to obtain a fused object feature.

[0108] In an embodiment of the present application, when the computer device obtains the fused object features, a cross-modal self-attention mechanism can be used. For example, for each object feature in the description information, the computer device obtains the similarity between the object feature and each text feature, and based on the obtained similarity, each text feature is fused with the object feature to obtain the fused object features, including: obtaining the query features, key features, and value features of each object feature and each text feature; for each object feature, based on the similarity between the query feature of the object feature and the key feature of each text feature, the value feature of the object feature is fused with the value feature of each text feature to obtain the fused object feature.

[0109] Optionally, the computer device determines the degree of association between each two fused object descriptor information based on multiple fused object descriptor information, including: based on each fused object feature, obtaining the query feature, key feature and value feature of the fused object feature, obtaining the similarity between the query feature of the fused object feature and the key feature of other fused object features other than the fused object feature among the multiple fused object features, and determining the degree of association between the fused object feature and the other fused object features based on the obtained similarity. For example, the computer device directly determines the obtained similarity as the degree of association between the fused object feature and the other fused object features; for another example, the computer device normalizes the obtained similarity to obtain the degree of association between the fused object feature and the other fused object features.

[0110] 503. The computer device weakens at least one object description sub-information in the description information based on the determined association degree to obtain processed description information.

[0111] The step 503 is similar to the above-mentioned step 303 and step 203, and will not be described in detail here.

[0112] 504. The computer device determines answer information of the question information based on the question information and the processed description information.

[0113] The step 504 is the same as the above step 204 and will not be described in detail here.

[0114] The answer information acquisition method provided in the embodiment of the present application takes into account that there is a lot of information in the target image that has nothing to do with the question information, and this information that is irrelevant to the question information can be regarded as noise information. Therefore, by determining the correlation between each object descriptor sub-information of the target image and the question information, the present application can find the object descriptor sub-information with a low correlation with the question information. Based on the determined correlation, at least one object descriptor sub-information in the description information is weakened, which can weaken the object descriptor sub-information with a low correlation with the question information, thereby achieving the effect of weakening the noise information in the target image and reducing the interference of the noise information, so that the answer information of the question information can be determined more accurately, thereby improving the accuracy of visual question answering.

[0115] Furthermore, the embodiments of the present application can discard object descriptor information of objects with low correlation with the question information. In this way, there is no need to pay attention to objects in the image that are not related to the question information, so that the content in the image can be more accurately focused, and the answer information of the question information can be more accurately determined, thereby improving the accuracy of visual question answering.

[0116] Figure 6 This is a flowchart of a method for obtaining answer information provided by an embodiment of the present application. This embodiment of the present application is illustrated by taking a computer device as an example of an execution subject. The embodiment includes:

[0117] 601. A computer device obtains description information of a target image based on a target image, where the description information includes relationship description information and object description information. The relationship description information includes multiple pieces of relationship description sub-information, and the object description information includes multiple pieces of object description sub-information.

[0118] 602. The computer device determines, based on the question information of the target image, a degree of association between each piece of relationship description sub-information in the relationship description information and the question information.

[0119] 603. The computer device weakens at least one piece of relationship description sub-information in the relationship description information based on the determined association degree to obtain processed relationship description information.

[0120] 604. The computer device determines the relevance between each object description sub-information in the object description information and the question information based on the question information of the target image.

[0121] 605. The computer device performs weakening processing on at least one object description sub-information in the object description information based on the determined association degree to obtain processed object description information.

[0122] 606. The computer device determines answer information of the question information based on the question information, the processed relationship description information and the processed object description information.

[0123] It should be noted that the above steps 601 to 603 are similar to the above steps 301 to 303, and will not be described one by one here. The above steps 604 to 606 are similar to the above steps 502 to 504, and will not be described one by one here.

[0124] Another point to be explained is that, in the embodiment of the present application, steps 602 to 603 may be performed first, and then steps 604 to 605 may be performed; steps 602 to 603 and steps 604 to 605 may also be performed simultaneously; steps 604 to 605 may also be performed first, and then steps 602 to 603 may be performed. The embodiment of the present application does not limit this.

[0125] The answer information acquisition method provided in the embodiment of the present application takes into account that there is a lot of information in the target image that has nothing to do with the question information, and these information that is irrelevant to the question information can be regarded as noise information. Therefore, the present application can find object descriptor information with low correlation with the question information by determining the correlation between each object descriptor sub-information of the target image and the question information. Based on the determined correlation, at least one object descriptor sub-information in the description information is weakened, and the object descriptor sub-information with low correlation with the question information can be weakened; by determining the correlation between each relationship descriptor sub-information of the target image and the question information, the relationship descriptor sub-information with low correlation with the question information can be found. Based on the determined correlation, at least one object descriptor sub-information in the description information is weakened, and the relationship descriptor sub-information with low correlation with the question information can be weakened, achieving the effect of weakening the noise information in the target image, reducing the interference of the noise information, so that the answer information of the question information can be determined more accurately, and the accuracy of visual question answering is improved.

[0126] It should be noted that, in a possible implementation, the relationship description sub-information refers to the relationship graph of the target image, a node in the relationship graph refers to an object in the target image, and the node feature of the node refers to the object feature of the object. The object description sub-information can refer to the object feature. Therefore, the description information includes the relationship description information, and the relationship description information includes the object description information. The embodiment of the present application can adopt Figure 3 The embodiment shown in the figure weakens at least one of the multiple relationship diagrams of the target image, and then uses Figure 5 The illustrated embodiment weakens the object features in the relationship graph, and then obtains answer information based on the weakened results.

[0127] For example, Figure 7As shown, the three relationship graphs are weightedly fused into a first target relationship graph according to the correlation between each relationship graph and the question information, and the object features with a low correlation with the question information are determined based on the correlation between the question information and the object features in the first target relationship graph. The nodes and edges corresponding to the object features are deleted in the first target relationship graph to obtain a second target relationship graph, and the answer information is obtained based on the second target relationship graph and the question information.

[0128] Another point that needs to be explained is that, in the embodiment of the present application, a method of “obtaining description information of the target image based on the target image” is also provided.

[0129] In a possible implementation, a computer device obtains description information of a target image based on a target image, including at least one of the following: (1) the description information includes relationship description information, the relationship description information includes multiple relationship description sub-information; based on the target image, the multiple relationship description sub-information of the target image is obtained. (2) the description information includes object description information, the object description information includes multiple object description sub-information, and different object description sub-information is used to describe different objects in the target image; based on the target image, the multiple object description sub-information of the target image is obtained.

[0130] Optionally, different relationship descriptor information is used to describe different relationships between objects in the target image; based on the target image, multiple relationship descriptor information of the target image is obtained, including at least one of the following: (1) the relationship description information includes semantic relationship descriptor information, which represents the semantic relationship between objects in the target image; based on the target image, the semantic relationship descriptor information of the target image is obtained; (2) the relationship description information includes spatial relationship descriptor information, which represents the spatial relationship between objects in the target image; based on the target image, the spatial relationship descriptor information of the target image is obtained; (3) the relationship description information includes implicit relationship descriptor information, which represents the implicit relationship between objects in the target image; based on the target image, the spatial implicit descriptor information of the target image is obtained.

[0131] The following is an example of a method for obtaining semantic relationship descriptor information. In a possible implementation, the semantic relationship descriptor information includes a semantic relationship tag; the computer device obtains the semantic relationship descriptor information of the target image based on the target image, including: processing the target image through a first relationship determination model to obtain at least one semantic relationship tag, the semantic relationship tag represents the semantic relationship between two objects in the target image, and the first relationship determination model is used to determine the semantic relationship tags for the two objects from multiple semantic relationship tags.

[0132] Among them, the first relationship determination model processes the target image, and for any two objects in the target image, the probability of each semantic relationship label in multiple semantic relationship labels can be obtained, and the semantic relationship label corresponding to the highest probability is determined as the semantic relationship label of the two objects.

[0133] In another possible implementation, the semantic relationship description sub-information refers to a first semantic relationship graph of the target image, a node in the first semantic relationship graph refers to an object in the target image, a node feature of the node refers to an object feature of the object, an edge between two nodes in the first semantic relationship graph refers to a semantic relationship between the corresponding two objects, an edge feature of the edge refers to a relationship feature of the semantic relationship between the corresponding two objects, and the direction of the edge is the direction indicated by the semantic relationship between the corresponding two objects. The computer device obtains the semantic relationship description sub-information of the target image based on the target image, including: processing the target image through the first relationship determination model to obtain at least one semantic relationship label, the semantic relationship label indicates the relationship between the two objects in the target image, the first relationship determination model is used to determine the semantic relationship label for the two objects from multiple semantic relationship labels; extracting features from each semantic relationship label and the semantic information of each object to obtain the relationship feature of each semantic relationship label and the object feature of each object; and determining the first semantic relationship graph of the target image based on the relationship feature of each semantic relationship label and the object feature of each object.

[0134] The semantic information of each object may be determined by an object recognition model. Optionally, the semantic information of the object may be the category information of the object. For example, the semantic information may be "man", "woman", "elderly", "child" and the like.

[0135] Optionally, the computer device determines a first semantic relationship graph of the target image based on the relationship features of each semantic relationship label and the object features of each object, including: determining the relationship features of each semantic relationship label as edge features of corresponding edges, determining the object features of each object as node features of corresponding nodes, and obtaining the first semantic relationship graph of the target image.

[0136] In another possible implementation, the semantic relationship description sub-information refers to a first semantic relationship graph of the target image, a node in the first semantic relationship graph refers to an object in the target image, a node feature of the node refers to an object feature of the object, an edge between two nodes in the first semantic relationship graph refers to a semantic relationship between the corresponding two objects, an edge feature of the edge refers to a relationship feature of the semantic relationship between the corresponding two objects, and the direction of the edge is the direction indicated by the semantic relationship between the corresponding two objects; the computer device obtains the semantic relationship description sub-information of the target image based on the target image, including: obtaining at least one semantic relationship label of the target image and semantic information of each object in the target image; performing feature extraction on the at least one semantic relationship label and the semantic information of each object through a first relationship determination model to obtain a relationship feature of each semantic relationship label and an object feature of each object; for each feature in the obtained relationship feature and object feature, according to the similarity between the feature and other features in the obtained feature except the feature, the other feature is merged with the feature to obtain an updated feature; and the first semantic relationship graph is determined based on each updated feature.

[0137] Optionally, the computer device determines the model through the first relationship, and for each feature in the obtained relationship features and object features, fuses the other features with the feature according to the similarity between the feature and other features in the obtained features except the feature, to obtain the updated feature, including: processing each feature in the obtained features through the first relationship determination model to obtain the query feature, key feature and value feature of the feature; and determining the model through the first relationship, for each feature in the obtained features, fuses the value feature of the feature with the value features of other features in the obtained features except the feature according to the similarity between the query feature of the feature and the key features of other features in the obtained features except the feature, to obtain the updated feature.

[0138] In some embodiments, a multi-head attention mechanism can be used when updating features. In one possible implementation, the first relationship determination model includes a feature extraction layer and a multi-head attention layer, and the multi-head attention layer includes multiple attention sub-layers; the first relationship determination model is used to extract features of the at least one semantic relationship label and the semantic information of each object to obtain the relationship features of each semantic relationship label and the object features of each object, including: extracting features of each semantic relationship label and the semantic information of each object through the feature extraction layer to obtain the relationship features of each semantic relationship label and the object features of each object, and the number of dimensions of the obtained features is the same as the number of the multiple attention sub-layers. The same; the model is determined through the first relationship, and for each feature in the obtained relationship features and object features, the other features are fused with the feature according to the similarity between the feature and other features in the obtained features except the feature, so as to obtain the updated feature, including: through the feature extraction layer, the sub-features belonging to the same dimension in each obtained feature are input into the corresponding attention sub-layer; through each attention sub-layer, for each input sub-feature, according to the similarity between the sub-feature and other sub-features in the input sub-feature except the sub-feature, the other sub-features are fused with the feature to obtain the updated sub-feature.

[0139] It should be noted that the embodiments of the present application are only exemplified by taking the example that the number of dimensions of each feature in the object feature and the relationship feature is the same as the number of the multiple attention sub-layers. In other embodiments, the number of dimensions of the relationship feature is the same as the number of the multiple attention sub-layers, and the number of dimensions of the object feature is greater than the number of the multiple attention sub-layers. In this case, the sub-features corresponding to the first dimension in the object feature and the relationship feature are input into the first attention sub-layer, and the sub-features corresponding to the second dimension in the object feature and the relationship feature are input into the second attention sub-layer, and so on. For the last attention sub-layer, the sub-features corresponding to the last dimension in the relationship feature are input into the last attention sub-layer, and the remaining input object feature sub-features in the object feature are input into the last attention sub-layer together.

[0140] In another possible implementation, the semantic relationship description sub-information refers to a second semantic relationship graph of the target image, a node in the second semantic relationship graph refers to an object in the target image, the node feature of the node refers to the object feature of the object, the edge between two nodes in the second semantic relationship graph refers to the association degree between the corresponding two objects, and the direction of the edge is the direction indicated by the association degree between the corresponding two objects. The computer device obtains the semantic relationship descriptor information of the target image based on the target image, including: obtaining at least one semantic relationship label of the target image and the semantic information of each object in the target image; performing feature extraction on the at least one semantic relationship label and the semantic information of each object through a first relationship determination model to obtain a relationship feature of each semantic relationship label and an object feature of each object; updating the object feature for each obtained object feature based on the object feature, other object features among the obtained object features except the object feature, and each relationship feature through the first relationship determination model to obtain an updated object feature; determining the association degree of the corresponding edge reference based on the two updated object features for every two updated object features through the first relationship determination model; and determining the second semantic relationship graph based on the updated object features and the association degree.

[0141] Optionally, the computer device determines the model through the first relationship, and for each object feature obtained, updates the object feature based on the object feature, other object features among the obtained object features except the object feature, and each relationship feature to obtain the updated object feature, including: processing each object feature through the first relationship determination model to obtain the query feature, key feature, and value feature of the object feature; determining the model through the first relationship, for each object feature, obtaining the product value of the query feature of the object feature and the key feature of each other object feature, obtaining the sum value of each product value and the corresponding relationship feature, normalizing each obtained sum value to obtain the weight of each other object feature, and based on the weight of each other object feature, fusing each other object feature with the object feature to obtain the updated object feature, wherein the relationship feature corresponding to the product value is the relationship feature corresponding to the two object features used to obtain the product value.

[0142] Optionally, the computer device determines the association degree of the corresponding edge reference based on the two updated object features through the first relationship determination model for every two updated object features, including: processing each updated object feature through the first relationship determination model to obtain the query feature, key feature and value feature of the updated object feature; determining the association degree of the corresponding edge reference based on the similarity between the query feature of one object feature and the key feature of another object feature through the first relationship determination model for every two updated object features, the corresponding edge being an edge from one object to another object, one of the objects being the object referred to by one of the object features, and the other object being the object referred to by the other object feature.

[0143] Optionally, the first relationship determination model includes a feature extraction layer and a multi-head attention layer, and the multi-head attention layer includes multiple attention sub-layers. The computer device performs feature extraction on the at least one semantic relationship tag and the semantic information of each object through the first relationship determination model to obtain the relationship feature of each semantic relationship tag and the object feature of each object, including: performing feature extraction on each semantic relationship tag and the semantic information of each object through the feature extraction layer to obtain the relationship feature of each semantic relationship tag and the object feature of each object, and the number of dimensions of the obtained features is the same as the number of the multiple attention sub-layers. The computer device determines the model through the first relationship, and for each object feature obtained, updates the object feature based on the object feature, other object features in the obtained object feature except the object feature, and each relationship feature to obtain an updated object feature, including: through the feature extraction layer, inputting relationship sub-features of different dimensions in each relationship feature into the corresponding attention sub-layer, and inputting object sub-features belonging to the same dimension in each object feature into the corresponding attention sub-layer; through each attention sub-layer, for each input object sub-feature, based on the object sub-feature, other object sub-features in the input object sub-feature except the object sub-feature, and the input relationship sub-feature, updates the object sub-feature to obtain an updated object sub-feature.

[0144] Optionally, the computer device determines the model through the first relationship, and for every two updated object features, determines the association degree of the corresponding edge reference based on the two updated object features, including: through each attention sub-layer, for every two updated object sub-features, determining the similarity of the two updated object sub-features as the sub-association degree of the corresponding edge reference; and fusing the sub-association degree of each edge determined by each attention sub-layer to obtain the association degree of each edge.

[0145] Optionally, the computer device determines the second semantic relationship graph based on the updated object features and the association degree, including: determining each updated object feature as a node feature of a corresponding node, determining each association degree as a association degree referred to by a corresponding edge, and obtaining a second semantic relationship graph of the target image.

[0146] Optionally, the computer device determines the second semantic relationship graph based on the updated object features and the association degree, including: continuing to update the updated object features and the association degree through the first relationship determination model until the number of updates reaches the target number; and determining the second semantic relationship graph based on the object features and association degree obtained from the last update.

[0147] Among them, the target number can be any positive integer greater than 1. The target number can be an empirical value, a value set by a technician, or a default value of the device. The embodiment of the present application does not limit the target number.

[0148] It should be noted that the method for obtaining the spatial relationship description information and the implicit relationship description is the same as the method for obtaining the semantic relationship description information, and will not be described in detail here.

[0149] For example, in one possible implementation, the spatial description sub-information includes a spatial relationship label, and the spatial relationship description sub-information of the target image is obtained based on the target image, including: determining a position condition satisfied by the position information of any two objects among the target objects based on the position information of the two objects; and determining the spatial relationship label corresponding to the position condition as the spatial relationship label of the two objects.

[0150] In a possible implementation, the spatial description sub-information refers to a first spatial relationship graph of the target image, a node in the first spatial relationship graph refers to an object in the target image, a node feature of the node refers to an object feature of the object, an edge between two nodes in the first spatial relationship graph refers to a spatial relationship between the corresponding two objects, an edge feature of the edge refers to a relationship feature of the spatial relationship between the corresponding two objects, and the direction of the edge is the direction indicated by the spatial relationship between the corresponding two objects. Based on the target image, the spatial relationship descriptor information of the target image is obtained, including: obtaining at least one spatial relationship label of the target image and the position information of each object in the target image; performing feature extraction on the obtained spatial relationship label and the position information of each object to obtain the relationship feature of each spatial relationship label and the object feature of each object; and determining the first spatial relationship graph of the target image based on the relationship feature of each spatial relationship label and the object feature of each object.

[0151] In one possible implementation, the first spatial relationship graph of the target image is determined based on the relationship features of each spatial relationship label and the object features of each object, including: determining the relationship features of each spatial relationship label as the edge features of the corresponding edges, determining the object features of each object as the node features of the corresponding nodes, and obtaining the first spatial relationship graph of the target image.

[0152] In a possible implementation, the spatial description sub-information refers to a first spatial relationship graph of the target image, a node in the first spatial relationship graph refers to an object in the target image, a node feature of the node refers to an object feature of the object, an edge between two nodes in the first spatial relationship graph refers to a spatial relationship between the corresponding two objects, an edge feature of the edge refers to a relationship feature of the spatial relationship between the corresponding two objects, and the direction of the edge is the direction indicated by the spatial relationship between the corresponding two objects. Based on the target image, the spatial relationship descriptor information of the target image is obtained, including: obtaining at least one spatial relationship label of the target image and position information of each object in the target image; performing feature extraction on the at least one spatial relationship label and the position information of each object through a third relationship determination model to obtain a relationship feature of each spatial relationship label and an object feature of each object; through the third relationship determination model, for each feature in the obtained relationship feature and object feature, according to the similarity between the feature and other features in the obtained feature except the feature, the other feature is merged with the feature to obtain an updated feature; based on each updated feature, the first spatial relationship graph is determined.

[0153] In a possible implementation, the model is determined through the third relationship, and for each feature in the obtained relationship features and object features, the other features are fused with the feature according to the similarity between the feature and other features in the obtained features except the feature, to obtain the updated feature, including: processing each feature in the obtained features through the third relationship determination model to obtain the query feature, key feature and value feature of the feature; and for each feature in the obtained features, the value feature of the feature is fused with the value feature of other features in the obtained features except the feature according to the similarity between the query feature of the feature and the key feature of other features in the obtained features except the feature, to obtain the updated feature.

[0154] In one possible implementation, the third relationship determination model includes a feature extraction layer and a multi-head attention layer, and the multi-head attention layer includes multiple attention sub-layers; the third relationship determination model is used to extract features of the at least one spatial relationship label and the position information of each object to obtain the relationship features of each spatial relationship label and the object features of each object, including: through the feature extraction layer, feature extraction is performed on each spatial system label and the position information of each object to obtain the relationship features of each spatial relationship label and the object features of each object, and the number of dimensions of the obtained features is equal to the number of the multiple attention sub-layers. The same; the model is determined through the third relationship, and for each feature in the obtained relationship features and object features, the other features are fused with the feature according to the similarity between the feature and other features in the obtained features except the feature, so as to obtain the updated feature, including: through the feature extraction layer, the sub-features belonging to the same dimension in each obtained feature are input into the corresponding attention sub-layer; through each attention sub-layer, for each input sub-feature, according to the similarity between the sub-feature and other sub-features in the input sub-feature except the sub-feature, the other sub-features are fused with the feature to obtain the updated sub-feature.

[0155] In one possible implementation, the spatial relationship descriptor information refers to a second spatial relationship graph of the target image, a node in the second spatial relationship graph refers to an object in the target image, the node feature of the node refers to the object feature of the object, the edge between two nodes in the second spatial relationship graph refers to the association degree between the corresponding two objects, and the direction of the edge is the direction indicated by the association degree between the corresponding two objects. Based on the target image, spatial relationship descriptor information of the target image is obtained, including: obtaining at least one spatial relationship label of the target image and the position information of each object in the target image; performing feature extraction on the at least one spatial relationship label and the position information of each object through a third relationship determination model to obtain a relationship feature of each spatial relationship label and an object feature of each object; for each object feature obtained, based on the object feature, other object features among the obtained object features except the object feature, and each relationship feature, updating the object feature through the third relationship determination model to obtain an updated object feature; for every two updated object features, based on the two updated object features, determining the association degree of the corresponding edge reference through the third relationship determination model; and determining the second spatial relationship graph based on the updated object features and the association degree.

[0156] In a possible implementation, the model is determined through the third relationship, and for each object feature obtained, based on the object feature, other object features among the obtained object features except the object feature, and each relationship feature, the object feature is updated to obtain the updated object feature, including: processing each object feature through the third relationship model to obtain the query feature, key feature, and value feature of the object feature; obtaining, for each object feature, through the third relationship model, the product value of the query feature of the object feature and the key feature of each other object feature, obtaining the sum value of each product value and the corresponding relationship feature, normalizing each obtained sum value to obtain the weight of each other object feature, and based on the weight of each other object feature, fusing each other object feature with the object feature to obtain the updated object feature, wherein the relationship feature corresponding to the product value is the relationship feature corresponding to the two object features used to obtain the product value.

[0157] In a possible implementation, the model is determined through the third relationship, and for every two updated object features, the association degree of the corresponding edge reference is determined based on the two updated object features, including: through the third relationship determination model, each updated object feature is processed to obtain the query feature, key feature and value feature of the updated object feature; through the third relationship determination model, for every two updated object features, based on the similarity between the query feature of one object feature and the key feature of the other object feature, the association degree of the corresponding edge reference is determined, the corresponding edge is an edge from one object to another object, the one object is the object referred to by the one object feature, and the other object is the object referred to by the other object.

[0158] In one possible implementation, the third relationship determination model includes a feature extraction layer and a multi-head attention layer, the multi-head attention layer includes multiple attention sub-layers; the third relationship determination model is used to extract features of the at least one spatial relationship label and the position information of each object to obtain the relationship features of each spatial relationship label and the object features of each object, including: the feature extraction layer is used to extract features of each spatial relationship label and the spatial information of each object to obtain the relationship features of each spatial relationship label and the object features of each object, and the number of dimensions of the obtained features is the same as the number of the multiple attention sub-layers; the third relationship determination model is used to obtain For each object feature of the obtained object feature, based on the object feature, other object features in the obtained object feature except the object feature, and each relationship feature, the object feature is updated to obtain an updated object feature, including: through the feature extraction layer, the relationship sub-features of different dimensions in each relationship feature are input into the corresponding attention sub-layer, and the object sub-features belonging to the same dimension in each object feature are input into the corresponding attention sub-layer; through each attention sub-layer, for each input object sub-feature, based on the object sub-feature, other object sub-features in the input object sub-feature except the object sub-feature, and the input relationship sub-feature, the object sub-feature is updated to obtain an updated object sub-feature.

[0159] In one possible implementation, the model is determined through the third relationship, and for every two updated object features, the correlation degree of the corresponding edge reference is determined based on the two updated object features, including: through each attention sub-layer, for every two updated object sub-features, based on the similarity of the two updated object sub-features, determining the sub-correlation degree of the corresponding edge reference; and fusing the sub-correlation degree of each edge determined by each attention sub-layer to obtain the correlation degree of each edge.

[0160] In one possible implementation, the second spatial relationship graph is determined based on the updated object features and the correlation degree, including: determining each updated object feature as a node feature of a corresponding node, determining each correlation degree as a correlation degree referred to by a corresponding edge, and obtaining a second spatial relationship graph of the target image.

[0161] In one possible implementation, the second spatial relationship graph is determined based on the updated object features and the correlation, including: continuing to update the updated object features and the correlation through the third relationship determination model until the number of updates reaches the target number; and determining the second spatial relationship graph based on the object features and correlation obtained from the last update.

[0162] In one possible implementation, the implicit relationship descriptor information refers to a first implicit relationship graph of the target image, a node in the first implicit relationship graph refers to an object in the target image, a node feature of the node refers to an object feature of the object, an edge between two nodes in the first implicit relationship graph refers to an implicit relationship between the corresponding two objects, an edge feature of the edge refers to a relationship feature of the implicit relationship between the corresponding two objects, and a direction of the edge is a direction indicated by the implicit relationship between the corresponding two objects. The method obtains spatial implicit descriptor information of the target image based on the target image, including: extracting features from semantic information and position information of each object through a second relationship determination model to obtain object features of each object; fusing the other object features with the object feature according to the similarity between the object feature and other object features other than the object feature among the obtained multiple object features through the second relationship determination model to obtain updated object features; determining the edge features corresponding to the two updated object features based on every two updated object features through the second relationship determination model, the corresponding edge features being the features of the edges between the two objects corresponding to the two updated object features; and determining the first implicit relationship graph based on each updated object feature and each edge feature.

[0163] In a possible implementation, the model is determined through the second relationship, and for each object feature, based on the similarity between the object feature and other object features other than the object feature in the multiple object features obtained, the other object features are merged with the object feature to obtain an updated object feature, including: determining the model through the second relationship, processing each object feature to obtain a query feature, a key feature, and a value feature of each object feature; determining the model through the second relationship, for each object feature, based on the similarity between the query feature of the object feature and the key features of other object features other than the object feature in the multiple object features obtained, the value feature of the object feature is merged with the value features of the other object features to obtain an updated object feature.

[0164] In one possible implementation, the model is determined through the second relationship, and based on every two updated object features, the edge features corresponding to the two updated object features are determined, including: the model is determined through the second relationship, and the fusion processing is performed on every two updated object features to obtain the edge features corresponding to the two updated object features.

[0165] In one possible implementation, the second relationship determination model includes a multi-head attention layer, which includes multiple attention sub-layers; through the second relationship determination model, for each object feature, according to the similarity between the object feature and other object features other than the object feature in the multiple object features obtained, the other object features are fused with the object feature to obtain an updated object feature, including: inputting object sub-features belonging to the same dimension in each object feature into the corresponding attention sub-layer; through each attention sub-layer, for each object sub-feature, according to the similarity between the object sub-feature and other object sub-features other than the object sub-feature in the input object sub-feature, the other object sub-features are fused with the object sub-feature to obtain an updated object sub-feature.

[0166] In a possible implementation, the implicit relationship descriptor information refers to a second implicit relationship graph of the target image, a node in the second implicit relationship graph refers to an object in the target image, a node feature of the node refers to an object feature of the object, an edge between two nodes in the second implicit relationship graph refers to a correlation between the two corresponding objects, and the direction of the edge is the direction indicated by the correlation between the two corresponding objects; based on the target image, obtaining the spatial implicit descriptor information of the target image includes: extracting features from the semantic information and position information of each object through a second relationship determination model to obtain the object feature of each object; updating the object feature for each object feature based on the object feature and other object features of the obtained multiple object features except the object feature through the second relationship determination model to obtain an updated object feature; determining the correlation referred to by the corresponding edge based on the two updated object features for every two updated object features through the second relationship determination model; and determining the second implicit relationship graph based on the updated object feature and the correlation.

[0167] In a possible implementation, the model is determined through the second relationship, and for each object feature, based on the object feature and other object features among the obtained multiple object features except the object feature, the object feature is updated to obtain the updated object feature, including: determining the model through the second relationship, processing each object feature to obtain the query feature, key feature and value feature of the object feature; determining the model through the second relationship, for each object feature, obtaining the product value of the query feature of the object feature and the key features of other object features, normalizing each obtained product value to obtain the weight of each other object feature, and fusing each other object feature with the object feature to obtain the updated object feature.

[0168] In a possible implementation, the model is determined through the second relationship, and for every two updated object features, the association degree of the corresponding edge reference is determined based on the two updated object features, including: processing each updated object feature through the second relationship determination model to obtain the query feature, key feature and value feature of the updated object feature; and determining the model through the second relationship, for every two updated object features, based on the similarity between the query feature of one of the object features and the key feature of the other object feature, the association degree of the corresponding edge reference is determined, the corresponding edge being an edge from one object to another object, the one of the objects being the object referred to by the one of the object features, and the other object being the object referred to by the other object feature.

[0169] In one possible implementation, the second relationship determination model includes a multi-head attention layer, which includes multiple attention sub-layers; the second relationship determination model is used to update, for each object feature, the object feature is updated based on the object feature and other object features other than the object feature in the multiple object features obtained to obtain the updated object feature, including: inputting object sub-features belonging to the same dimension in each object feature into the corresponding attention sub-layer; through each attention sub-layer, for each input object sub-feature, the object sub-feature is updated based on the object sub-feature and other object sub-features other than the object sub-feature in the input object sub-feature to obtain the updated object sub-feature.

[0170] In one possible implementation, the model is determined through the second relationship, and for every two updated object features, the correlation degree of the corresponding edge reference is determined based on the two updated object features, including: through each attention sub-layer, for every two updated sub-features of the object, based on the similarity of the two updated sub-features of the object, determining the sub-correlation degree of the corresponding edge reference; and fusing the sub-correlation degree of each edge determined by each attention sub-layer to obtain the correlation degree of each edge.

[0171] In one possible implementation, the second implicit relationship graph is determined based on the updated object features and the association degree, including: determining each updated object feature as a node feature of a corresponding node, determining each association degree as a association degree referred to by a corresponding edge, and obtaining a second implicit relationship graph of the target image.

[0172] In one possible implementation, the second implicit relationship graph is determined based on the updated object features and the association degree, including: continuing to update the updated object features and the association degree through the second relationship determination model until the number of updates reaches the target number; and determining the second implicit relationship graph based on the object features and association degree obtained from the last update.

[0173] Another point that needs to be explained is that after obtaining the semantic relationship graph, spatial relationship graph and implicit relationship graph, the semantic relationship graph, spatial relationship graph and implicit relationship graph can be fused to obtain the target relationship graph, and the node features in the target relationship graph can be used as object descriptor information.

[0174] Figure 8 This is a schematic diagram of the structure of an answer information acquisition device provided in an embodiment of the present application, see Figure 8 The device includes: an information acquisition module 801, which is used to acquire description information of the target image based on the target image, and the description information includes multiple description sub-information; a correlation determination module 802, which is used to determine the correlation between each description sub-information in the description information and the question information based on the question information of the target image; a processing module 803, which is used to weaken at least one description sub-information in the description information based on the determined correlation to obtain processed description information; an answer determination module 804, which is used to determine the answer information of the question information based on the question information and the processed description information.

[0175] like Fig. 9 As shown, in a possible implementation, the description information includes relationship description information, the relationship description information includes multiple relationship description sub-information, and different relationship description sub-information is used to describe different relationships between objects in the target image; the problem information is used to represent the problem associated with the object in the target image; the correlation determination module 802 is used to determine the correlation between each relationship description sub-information in the relationship description information and the problem information based on the problem information, and the correlation represents the degree of correlation between the relationship represented by the relationship description sub-information and the problem information.

[0176] In a possible implementation, the relationship descriptor information refers to a relationship graph of the target image, and different relationship descriptor information refers to different types of relationship graphs of the target image; a node in the relationship graph refers to an object in the target image, and the node feature of the node refers to the object feature of the object; the edge between two nodes in the relationship graph refers to the relationship between the corresponding two objects, the edge feature of the edge refers to the relationship feature of the relationship between the corresponding two objects, and the direction of the edge is the direction indicated by the relationship between the two objects; or, the edge between two nodes in the relationship graph refers to the correlation under the corresponding relationship between the two objects, and the direction of the edge is the direction indicated by the correlation between the corresponding two objects; the correlation determination module 802 is used to determine the first global feature of each relationship graph of the multiple relationship graphs of the target image; determine the second global feature of the question information; and determine the similarity between the first global feature of each relationship graph and the second global feature as the correlation between each relationship graph and the question information.

[0177] In one possible implementation, the processing module 803 includes: a weight determination unit 8031, which is used to determine the weight of each relationship graph based on the determined association degree; a fusion unit 8032, which is used to weightedly fuse the first global features of the multiple relationship graphs based on the weight of each relationship graph, so as to obtain the third global feature of the target image, if the processed description information includes the third global feature of the target image; or, if the processed description information includes the target relationship graph of the target image, and weightedly fuse the multiple relationship graphs based on the weight of each relationship graph, so as to obtain the target relationship graph of the target image.

[0178] In one possible implementation, the description information includes object description information, the object description information includes multiple object description sub-information, different object description sub-information is used to describe different objects in the target image, and the question information is used to represent the problem associated with the object in the target image; the correlation determination module 802 is used to determine the correlation between each object description sub-information in the object description information and the question information based on the question information, and the correlation is used to represent the degree of correlation between the object represented by the object description sub-information and the question information.

[0179] In one possible implementation, the correlation determination module 802 includes: a fusion unit 8021, used to fuse each object description sub-information in the description information with the question information to obtain multiple fused object description sub-information; a determination unit 8022, used to determine the correlation between each two fused object description sub-information based on the multiple fused object description sub-information; the determination unit 8022 is also used to determine the correlation between each fused object description sub-information and the question information based on the correlation and value of other fused object description sub-information in the multiple fused object description sub-information except the fused object description sub-information and the fused object description sub-information.

[0180] In one possible implementation, the object description sub-information refers to object features; the fusion unit 8021 is used to obtain text features of each word in the question information; for each object feature in the description information, the similarity between the object feature and each text feature is obtained, and based on the obtained similarity, each text feature is fused with the object feature to obtain a fused object feature.

[0181] In one possible implementation, the object descriptor information refers to an object feature; the determination unit 8022 is used to obtain, based on each fused object feature, a query feature, a key feature and a value feature of the fused object feature, obtain a similarity between the query feature of the fused object feature and a key feature of other fused object features among multiple fused object features except the fused object feature, and determine, based on the obtained similarity, a correlation between the fused object feature and the other fused object features.

[0182] In a possible implementation, the processing module 803 is used to weaken at least one fused object description sub-information in the description information based on the determined correlation degree to obtain the processed description information.

[0183] In one possible implementation, the processing module 803 is used to determine the weight of each descriptor information based on the determined degree of association, and weaken the multiple descriptor information based on the weight of each descriptor information to obtain the processed description information; or, the processing module 803 is used to determine at least one descriptor information from the description information based on the determined degree of association, weaken the at least one descriptor information, and obtain the processed description information.

[0184] In one possible implementation, the processing module 803 is used to determine the weight of each descriptor information in the at least one descriptor information based on the determined correlation, and based on the weight of each descriptor information, weaken the at least one descriptor information in the description information to obtain the processed description information; or, the processing module 803 is used to delete the at least one descriptor information from the description information to obtain the processed description information.

[0185] It should be noted that: the answer information acquisition device provided in the above embodiment only uses the division of the above functional modules as an example when recommending information. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the answer information acquisition device provided in the above embodiment and the answer information acquisition method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0186] In an exemplary embodiment, a computer device is provided, which includes one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the answer information acquisition method as in the above-mentioned embodiment.

[0187] Optionally, the computer device is provided as a terminal. Fig.10 The structure block diagram of a terminal 1000 provided by an exemplary embodiment of the present application is shown. The terminal 1000 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), an MP4 player (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4), a laptop computer or a desktop computer. The terminal 1000 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0188] The terminal 1000 includes: a processor 1001 and a memory 1002 .

[0189] The processor 1001 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1001 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1001 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1001 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0190] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one program code, which is executed by the processor 1001 to implement the answer information acquisition method provided in the method embodiment of the present application.

[0191] In some embodiments, the terminal 1000 may further optionally include: a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002 and the peripheral device interface 1003 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1003 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1004, a display screen 1005, a camera 1006, an audio circuit 1007, a positioning component 1008 and a power supply 1009.

[0192] The peripheral device interface 1003 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002, and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002, and the peripheral device interface 1003 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0193] The radio frequency circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1004 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1004 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0194] The display screen 1005 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 also has the ability to collect touch signals on the surface or above the surface of the display screen 1005. The touch signal can be input to the processor 1001 as a control signal for processing. At this time, the display screen 1005 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1005 can be one, and the front panel of the terminal 1000 is set; in other embodiments, the display screen 1005 can be at least two, which are respectively set on different surfaces of the terminal 1000 or are folded; in some other embodiments, the display screen 1005 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 1000. Even, the display screen 1005 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1005 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0195] The camera assembly 1006 is used to capture images or videos. Optionally, the camera assembly 1006 includes a front camera and a rear camera. The front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1006 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0196] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1001 for processing, or input them into the radio frequency circuit 1004 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1000. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1007 may also include a headphone jack.

[0197] The positioning component 1008 is used to locate the current geographical location of the terminal 1000 to implement navigation or LBS (Location Based Service). The positioning component 1008 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system or the European Union's Galileo system.

[0198] The power supply 1009 is used to power various components in the terminal 1000. The power supply 1009 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1009 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0199] In some embodiments, the terminal 1000 further includes one or more sensors 1010 , including but not limited to: an acceleration sensor 1011 , a gyroscope sensor 1012 , a pressure sensor 1013 , a fingerprint sensor 1014 , an optical sensor 1015 , and a proximity sensor 1016 .

[0200] Those skilled in the art will understand that Fig.10 The structure shown in the figure does not constitute a limitation on the terminal 1000, and the terminal 1000 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0201] Optionally, the computer device is provided as a server. Fig.11This is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 1100 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1101 and one or more memories 1102, wherein the memory 1102 stores at least one program code, and the at least one program code is loaded and executed by the processor 1101 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be described in detail here.

[0202] The server 1100 is used to execute the steps executed by the server in the above method embodiment.

[0203] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a program code, and the program code can be executed by a processor in a computer device to complete the answer information acquisition method in the above embodiment. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0204] In an exemplary embodiment, a computer program or a computer program product is also provided. The computer program or the computer program product includes a computer program code. When the computer program code is executed by a computer, the computer implements the answer information acquisition method in the above embodiment.

[0205] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0206] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for obtaining answer information, characterized in that: The method comprises: based on a target image, obtaining description information of the target image, the description information comprising a plurality of descriptor information; based on question information of the target image, determining the relevance between each descriptor information in the description information and the question information; based on the determined relevance, performing a weakening process on at least one descriptor information in the description information to obtain processed description information; based on the question information and the processed description information, determining answer information of the question information; The description information includes object description information, the object description information includes a plurality of object description sub-information, different object description sub-information is used to describe different objects in the target image, and the question information is used to represent a question associated with the object in the target image; the determining the degree of association between each description sub-information in the description information and the question information based on the question information of the target image comprises: determining the degree of association between each object description sub-information in the object description information and the question information based on the question information, the degree of association being used to represent the degree of association between the object represented by the object description sub-information and the question information; The determining, based on the question information, the degree of association between each object description sub-information in the object description information and the question information comprises: fusing each object description sub-information in the description information with the question information to obtain a plurality of fused object description sub-information; determining, based on the plurality of fused object description sub-information, the degree of association between every two fused object description sub-information; and determining, for each fused object description sub-information, the degree of association between the fused object description sub-information and the question information based on the degree of association and value of other fused object description sub-information in the plurality of fused object description sub-information except the fused object description sub-information.

2. The method according to claim 1, characterized in that The description information includes relationship description information, the relationship description information includes a plurality of relationship description sub-information, and different relationship description sub-information is used to describe different relationships between objects in the target image; the question information is used to represent questions associated with the objects in the target image; The determining of the correlation between each descriptor sub-information in the description information and the question information based on the question information of the target image includes: determining the correlation between each relationship description sub-information in the relationship description information and the question information based on the question information, wherein the correlation indicates the degree of correlation between the relationship represented by the relationship description sub-information and the question information.

3. The method according to claim 2, characterized in that The relationship descriptor information refers to a relationship graph of the target image, and different relationship descriptor information refers to different types of relationship graphs of the target image; a node in the relationship graph refers to an object in the target image, and a node feature of the node refers to an object feature of the object; The edge between two nodes in the relationship graph refers to the relationship between the corresponding two objects, the edge feature of the edge refers to the relationship feature of the relationship between the corresponding two objects, and the direction of the edge is the direction indicated by the relationship between the two objects; or, the edge between two nodes in the relationship graph refers to the degree of association under the corresponding relationship between the corresponding two objects, and the direction of the edge is the direction indicated by the degree of association between the corresponding two objects; The step of determining the degree of association between each piece of relationship description sub-information in the relationship description information and the question information based on the question information includes: determining a first global feature of each relationship graph among multiple relationship graphs of the target image; determining a second global feature of the question information; and determining the similarity between the first global feature of each relationship graph and the second global feature as the degree of association between each relationship graph and the question information.

4. The method according to claim 3, characterized in that The weakening process of at least one descriptor information in the description information based on the determined degree of association to obtain the processed description information includes: determining the weight of each relationship graph based on the determined degree of association; the processed description information includes the third global feature of the target image, and based on the weight of each relationship graph, the first global features of the multiple relationship graphs are weightedly fused to obtain the third global feature of the target image; or, the processed description information includes the target relationship graph of the target image, and based on the weight of each relationship graph, the multiple relationship graphs are weightedly fused to obtain the target relationship graph of the target image.

5. The method according to claim 1, characterized in that The object description sub-information refers to the object feature; and fusing each object description sub-information in the description information with the question information to obtain multiple fused object description sub-information, including: obtaining the text feature of each word in the question information; for each object feature in the description information, obtaining the similarity between the object feature and each text feature, and based on the obtained similarity, fusing each text feature with the object feature to obtain the fused object feature.

6. The method according to claim 5, characterized in that The determining the association between each two fused object descriptor sub-information based on the multiple fused object descriptor sub-information includes: based on each fused object feature, obtaining the query feature, key feature and value feature of the fused object feature, obtaining the similarity between the query feature of the fused object feature and the key feature of other fused object features among the multiple fused object features except the fused object feature, and determining the association between the fused object feature and the other fused object features based on the obtained similarity.

7. The method according to claim 1, characterized in that The weakening process of at least one descriptor information in the description information based on the determined correlation degree to obtain the processed description information includes: weakening the at least one fused object descriptor information in the description information based on the determined correlation degree to obtain the processed description information.

8. The method according to claim 1, characterized in that The weakening processing of at least one descriptor information in the description information based on the determined association degree to obtain the processed description information includes: determining the weight of each descriptor information based on the determined association degree, and weakening the multiple descriptor information based on the weight of each descriptor information to obtain the processed description information; or determining at least one descriptor sub-information from the description information based on the determined association degree, and weakening the at least one descriptor sub-information to obtain the processed description information.

9. The method according to claim 8, characterized in that The weakening processing of the at least one descriptor information to obtain the processed description information includes: determining the weight of each descriptor information in the at least one descriptor information based on the determined association degree, and weakening the at least one descriptor information in the description information based on the weight of each descriptor information to obtain the processed description information; or deleting the at least one descriptor information from the description information to obtain the processed description information.

10. A device for obtaining answer information, characterized in that: The device comprises: an information acquisition module, used to acquire description information of the target image based on the target image, wherein the description information comprises a plurality of description sub-information; a correlation determination module, used to determine the correlation between each description sub-information in the description information and the question information based on the question information of the target image; a processing module, used to weaken at least one description sub-information in the description information based on the determined correlation to obtain processed description information; an answer determination module, used to determine answer information of the question information based on the question information and the processed description information; The description information includes object description information, the object description information includes a plurality of object description sub-information, different object description sub-information is used to describe different objects in the target image, and the question information is used to represent a question associated with the object in the target image; the determining the degree of association between each description sub-information in the description information and the question information based on the question information of the target image comprises: determining the degree of association between each object description sub-information in the object description information and the question information based on the question information, the degree of association being used to represent the degree of association between the object represented by the object description sub-information and the question information; The determining, based on the question information, the degree of association between each object description sub-information in the object description information and the question information comprises: fusing each object description sub-information in the description information with the question information to obtain a plurality of fused object description sub-information; determining, based on the plurality of fused object description sub-information, the degree of association between every two fused object description sub-information; and determining, for each fused object description sub-information, the degree of association between the fused object description sub-information and the question information based on the degree of association and value of other fused object description sub-information in the plurality of fused object description sub-information except the fused object description sub-information.

11. A computer device, characterized in that: The computer device includes one or more processors and one or more memories, wherein the one or more memories store at least one program code, and the at least one program code is loaded and executed by the one or more processors to implement the operations performed by the answer information acquisition method as described in any one of claims 1 to claim 9.

12. A computer-readable storage medium, characterized in that: The storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed by the answer information acquisition method according to any one of claims 1 to 9.

Images