A knowledge-oriented pointer representation understanding method, device and storage medium

Abstract

The application discloses a knowledge-oriented pointing representation understanding method and device and a storage medium, and the method comprises the following steps: according to the correlation between image visual features and pointing representation sentence text features, the pointing representation sentence is divided into visual correlation sentence segments and knowledge correlation sentence segments; knowledge retrieval is carried out according to the knowledge correlation sentence segments, so as to obtain the category name of an object corresponding to the knowledge; and the visual correlation sentence segments and the category name of the object corresponding to the knowledge are subjected to feature fusion, so as to predict the boundary box coordinates of the pointing object. The application solves the problem that the current knowledge-oriented pointing representation understanding technology is interfered by irrelevant information in the pointing representation sentence and the knowledge, so as to cause the related knowledge retrieval error and the pointing object positioning error, can improve the accuracy of the related knowledge retrieval, and can improve the object positioning accuracy of the knowledge-oriented pointing representation understanding. The application can be widely applied to the technical field of computer vision.

Description

Technical Field

[0001] The present invention relates to the fields of multimodality, natural language processing and computer vision technology, and in particular to a knowledge-oriented referential expression understanding method, device and storage medium. Background Art

[0002] Referential expression understanding is a technology that locates the visual objects referred to in an image by understanding the intent of referential expressions. Referential expressions are expressed in natural human language, so this technology falls under the purview of natural language processing; visual objects exist in the form of images, so this technology also falls under the purview of computer vision. Referential expression understanding is a foundational technology for multimodal language-vision scenarios and has a wide range of applications, assisting downstream applications such as robotic manipulation, visual intelligent question answering, and visual language navigation.

[0003] Knowledge-based referential expression understanding technology extends traditional referential expression understanding technology to the knowledge domain. This technology locates the visual object referred to in an image by understanding the knowledge implicit in referential expression sentences. A key difference between this and traditional referential expression understanding technology is that referential expressions reference external knowledge, such as the functional attributes of the visual object. This requires knowledge-based referential expression understanding technology to retrieve relevant knowledge and infer the final referent.

[0004] However, existing knowledge-oriented referential expression understanding technologies are easily interfered with by irrelevant information in referential expression sentences during knowledge retrieval, resulting in the retrieval of knowledge irrelevant to the referent object; when reasoning knowledge to locate objects, they are easily interfered with by irrelevant information in the knowledge, resulting in the location of the wrong visual object. Summary of the Invention

[0005] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a knowledge-oriented referential expression understanding method, device and storage medium.

[0006] The technical solution adopted in the present invention is:

[0007] A knowledge-oriented referential expression understanding method includes the following steps:

[0008] Construct a knowledge-based referential expression understanding dataset; each sample in the dataset consists of an image, a referential expression sentence involving object knowledge, a set of bounding box coordinates of the correct referent object, and a sentence fragment related to the correct knowledge;

[0009] Input the referential expression sentence of each sample into the text encoder and output the corresponding sentence text features; input the image of each sample into the visual encoder and output the corresponding image visual features;

[0010] Calculate the correlation between the image visual features and the sentence text features, and output a visual relevance score for each word in the sentence; splice words whose visual relevance scores are less than or equal to a first preset threshold into knowledge-related sentence fragments, and splice words whose visual relevance scores are greater than the first preset threshold into visually related sentence fragments;

[0011] Input the knowledge-related sentence fragment into the knowledge retrieval model, and output the category name of the object described by the knowledge-related sentence fragment; input the object category name into the text encoder, and output the corresponding object category name text feature;

[0012] The visually relevant sentence fragments are fused with the text features of the object category name, and the fused features are input into the visual object localization model to output the predicted bounding box coordinates of the referred object.

[0013] Furthermore, the text encoder includes RNN, a variant of RNN or BERT series;

[0014] The visual encoder includes ResNet series, DarkNet series or DETR series;

[0015] The knowledge retrieval model includes RNN, RNN variants or BERT series;

[0016] The visual object positioning model includes VLTVG, a variant of VLTVG, Transformer or a variant of Transformer.

[0017] Furthermore, in the step of calculating the correlation between the image visual features and the sentence text features, the calculation objects include global image visual features, local image visual features, and global sentence text features, and local sentence text features; wherein the local image visual features include the visual features of the region in the image and the object bounding box, and the local sentence text features include the text features of the clauses and words in the sentence;

[0018] The calculation method used is any one of Euclidean distance, cosine distance, Jaccard distance, matrix multiplication, a variant of matrix multiplication, attention mechanism, or a variant of attention mechanism;

[0019] The visual relevance score of each word in the output sentence includes:

[0020] Get the softmax, maximum, average, and tertiary values ​​in the correlation results as the visual relevance score for each word in the sentence.

[0021] Furthermore, the calculation of the correlation between the image visual features and the sentence text features and outputting the visual correlation score of each word in the sentence includes:

[0022] Iterate T rounds to decompose the text feature vector x into multiple clauses and obtain the clause text feature vector x sub(T) , specifically expressed as:

[0023] x sub(T) =s sub(T) x,

[0024] s sub(T) =Conv(v sub(T-1) ·x·s sub(T-1) )

[0025] Where s sub(T-1) It is a score vector for calculating the clause position, Conv() is a convolution operation, and the clause visual feature vector of the Tth round is v sub(T) , specifically expressed as:

[0026] v sub(T) =ReLU(v sub(T-1) Linear(x sub(T) )+Linear(x sub(T) ))

[0027] In the formula, ReLU() is the activation function and Linear() is the linear function;

[0028] Merge the clause text feature vector and clause visual feature vector obtained in round T to obtain the correlation x′ between the visual feature vector v and the text feature vector x sub , specifically expressed as:

[0029]

[0030] The correlation x′ sub Mapped to the visual relevance score x for each word in the sentence vi , specifically expressed as:

[0031] x vi =Softmax(Linear(x′ sub ))·x+x.

[0032] Furthermore, the step of inputting the knowledge-related sentence fragment into the knowledge retrieval model and outputting the category name of the object described by the knowledge-related sentence fragment; inputting the object category name into the text encoder and outputting the corresponding object category name text features includes:

[0033] The knowledge-related sentence fragment S kn As a prompt word, input into the knowledge retrieval model E k Perform mask prediction in , obtain the category name of the object described by the sentence fragment related to the knowledge, and obtain the category name z of the top M objects that match best;

[0034] Input the category name z of the top M most matching objects into the text encoder E t In the output, the corresponding object category name text feature x z(M) .

[0035] Furthermore, in the step of fusing the visually related sentence fragments with the text features of the object category name, the fusion operation includes item-by-item multiplication, item-by-item addition, feature connection, attention mechanism or a variant of the attention mechanism.

[0036] Furthermore, the fusion operation of visually related sentence fragments and object category name text features includes:

[0037] The object category name text feature x z(M) Splice to sentence fragment S vi The first, last and middle parts of the concatenated vectors are then linearly mapped to the knowledge-related sentence features y M ;

[0038] The fused features are input into the visual object localization model to output the predicted bounding box coordinates of the object, including:

[0039] Iterate M rounds to transform the knowledge related sentence features y M Fused with the visual feature vector v to obtain the fused feature v kn , the fused feature v kn Input to the visual object localization model E g In the example, we get the predicted bounding box coordinates B of the object being referred to. pred .

[0040] Furthermore, the loss function of the visual object positioning model training is expressed as follows:

[0041]

[0042] in, is the error between the knowledge-related sentence fragment and the correct knowledge-related sentence fragment, is the error between the predicted bounding box coordinates of the referenced object and the bounding box coordinates of the correct referenced object, is the error between the predicted bounding box coordinates of the reference object and the correct bounding box coordinates of the reference object, and λ1, λ2 and λ3 are adjustment factors used to balance the loss function.

[0043] Another technical solution adopted in the present invention is:

[0044] A knowledge-oriented referential expression understanding device, comprising:

[0045] at least one processor;

[0046] at least one memory for storing at least one program;

[0047] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0048] Another technical solution adopted in the present invention is:

[0049] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to perform the method described above when executed by the processor.

[0050] The beneficial effect of the present invention is: the present invention utilizes a core method based on sentence fragments to solve the problem that the current knowledge-oriented referential expression understanding technology is interfered with by irrelevant information in referential expression sentences and knowledge, thereby leading to errors in related knowledge retrieval and reference object positioning. It can improve the accuracy of related knowledge retrieval and the accuracy of object positioning in knowledge-oriented referential expression understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0052] Figure 1 It is a flowchart of a knowledge-oriented referential expression understanding method in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0054] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0055] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0056] Furthermore, in the description of this invention, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0057] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0058] like Figure 1 As shown, this embodiment provides a knowledge-oriented referential expression understanding method, and the specific steps include:

[0059] S1. Data Acquisition: Obtain a knowledge-oriented referential expression understanding dataset. Each sample in the dataset consists of an image I, a referential expression sentence S involving object knowledge, a set of bounding box coordinates B that correctly refer to the object, and a sentence fragment K related to the correct knowledge.

[0060] S2, feature encoding: Input the text content of each sample's referential expression sentence S into the text encoder E t Output the corresponding text feature vector x; input the image I of each sample into the visual encoder E v , output the corresponding visual feature vector v.

[0061] Build a text encoder E t , used to extract the text features of the referential expression sentence S. Construct a visual encoder E v , used to extract the visual features of image I. Among them, the text encoder E t Including RNN, RNN variants or BERT series; visual encoder E v Including ResNet series, DarkNet series or DETR series.

[0062] In an optional embodiment, the text encoder E t is BERT-base; visual encoder E v It is DETR-ResNet101.

[0063] S3. Sentence fragment division: Calculate the correlation between the visual feature vector v and the text feature vector x, and output the visual relevance score of each word in the sentence; splice words with a visual relevance score less than or equal to a first preset threshold into knowledge-related sentence fragments, and splice words with a visual relevance score greater than the first preset threshold into visually related sentence fragments.

[0064] The calculation objects include global image visual features, local image visual features (including visual features of regions and object bounding boxes in the image), global sentence text features, and local sentence text features (including text features of clauses and words in the sentence). In an optional embodiment, the calculation method used includes Euclidean distance, cosine distance, Jaccard distance, matrix multiplication, a variant of matrix multiplication, an attention mechanism, or a variant of an attention mechanism; the visual relevance score of each word in the output sentence includes obtaining the softmax, maximum value, average value, and tertile in the correlation result as the visual relevance score of each word in the sentence.

[0065] In an optional embodiment, the text feature vector x is decomposed into three clauses by iterating T rounds to obtain the clause text feature vector x sub(T) , specifically expressed as:

[0066] x sub(T) =s sub(T) x,

[0067] s sub(T) =Conv(v sub(T-1) ·x·ssub(T-1) )

[0068] Among them, s sub(T-1) It is a score vector for calculating the clause position, Conv() is a convolution operation, and the clause visual feature vector of the Tth round is v sub(T) , specifically expressed as:

[0069] v sub(T) =ReLU(v sub(T-1) Linear(x sub(T) )+Linear(x sub(T) )),

[0070] Among them, ReLU() is the activation function and Linear() is the linear function;

[0071] Combine the clause text feature vector and clause visual feature vector obtained in round T to obtain the correlation x′ between the visual feature vector v and the text feature vector x sub , specifically expressed as:

[0072]

[0073] The correlation x′ sub Mapped to the visual relevance score x for each word in the sentence vi , specifically expressed as:

[0074] x vi =Softmax(Linear(x′ sub ))·x+x.

[0075] The visual correlation score x vi Words less than or equal to the threshold θ are concatenated into knowledge-related sentence fragments S kn , the visual correlation score x vi Words greater than the threshold θ are concatenated into visually related sentence fragments S vi .

[0076] S4. Knowledge retrieval: Building a knowledge retrieval model E k ; Input the knowledge-related sentence fragment into the knowledge retrieval model, and output the category name of the object described by the knowledge-related sentence fragment; input the object category name into the text encoder, and output the corresponding object category name text feature.

[0077] In an optional embodiment, the knowledge-related sentence fragment S kn Input as prompt words into the knowledge retrieval model E k Perform mask prediction in , get the category name of the object described by the sentence fragment related to the knowledge, and get the top M items with the best match as z;

[0078] Input the category name z of the top M most matching objects into the text encoder E t In the output, the corresponding object category name text feature x z(M) .

[0079] The knowledge retrieval model E k Including RNN, RNN variants or BERT series. In an optional embodiment, the knowledge retrieval model E k It is BERT-large.

[0080] S5. Visually related sentence fragments S vi Text features x with object category names z(M) Implement fusion operations, including term-by-term multiplication, term-by-term addition, feature concatenation, attention mechanism, and its variants.

[0081] In an optional embodiment, x z(M) Vector splicing to S vi The first, last and middle parts of the concatenated vectors are then linearly mapped to the knowledge-related sentence features y M .

[0082] S6, object positioning: iterate M rounds to transform the knowledge-related sentence features y M Fused with the visual feature vector v to obtain the fused feature v kn , the fused feature v kn Input to the visual object localization model E g In the example, we get the predicted bounding box coordinates B of the object being referred to. pred .

[0083] In an optional embodiment, the visual object positioning model E g For VLTVG and Transformer decoder, the loss function of model training is expressed as:

[0084]

[0085] in, S is a knowledge-related sentence fragment kn The MSE error between the sentence fragments K associated with the correct knowledge, B is the predicted bounding box coordinate of the object pred The L1 error between the bounding box coordinates B and the correct object, B is the predicted bounding box coordinate of the object pred The GIoU error between the bounding box coordinates B and the correct object, λ1, λ2 and λ3 are adjustment factors used to balance the loss function.

[0086] This embodiment further provides a knowledge-oriented referential expression understanding device, including:

[0087] at least one processor;

[0088] at least one memory for storing at least one program;

[0089] When the at least one program is executed by the at least one processor, the at least one processor implements the following Figure 1 The method shown.

[0090] A knowledge-oriented referential expression understanding device of this embodiment can execute a knowledge-oriented referential expression understanding method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0091] The present application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 1 The method shown.

[0092] This embodiment also provides a storage medium that stores instructions or programs that can execute a knowledge-oriented reference expression understanding method provided by the method embodiment of the present invention. When the instructions or program are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0093] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0094] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0095] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0096] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0097] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0098] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0099] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0100] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0101] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A knowledge-oriented referential expression comprehension method, characterized in that: The following steps are involved: Construct a knowledge-based referential expression understanding dataset; each sample in the dataset consists of an image, a referential expression sentence involving object knowledge, a set of bounding box coordinates of the correct referent object, and a sentence fragment related to the correct knowledge; Input the referential expression sentence of each sample into the text encoder and output the corresponding sentence text features; Input the image of each sample into the visual encoder and output the corresponding image visual features; Calculate the correlation between the image visual features and the sentence text features, and output a visual relevance score for each word in the sentence; splice words whose visual relevance scores are less than or equal to a first preset threshold into knowledge-related sentence fragments, and splice words whose visual relevance scores are greater than the first preset threshold into visually related sentence fragments; Input the knowledge-related sentence fragment into the knowledge retrieval model, and output the category name of the object described by the knowledge-related sentence fragment; input the object category name into the text encoder, and output the corresponding object category name text feature; The visually relevant sentence fragments are fused with the text features of the object category name, and the fused features are input into the visual object localization model to output the predicted bounding box coordinates of the referred object.

2. A knowledge-oriented referential expression understanding method according to claim 1, characterized in that: The text encoder includes RNN, a variant of RNN or the BERT series; The visual encoder includes ResNet series, DarkNet series or DETR series; The knowledge retrieval model includes RNN, RNN variants or BERT series; The visual object positioning model includes VLTVG, a variant of VLTVG, Transformer or a variant of Transformer.

3. A knowledge-oriented referential expression comprehension method according to claim 1, characterized in that: In the step of calculating the correlation between the image visual features and the sentence text features, the calculation objects include global image visual features, local image visual features, global sentence text features, and local sentence text features; wherein the local image visual features include visual features of regions and object bounding boxes in the image, and the local sentence text features include text features of clauses and words in the sentence; The calculation method used is any one of Euclidean distance, cosine distance, Jaccard distance, matrix multiplication, a variant of matrix multiplication, attention mechanism, or a variant of attention mechanism; The visual relevance score of each word in the output sentence includes: Get the softmax, maximum, average, and tertiary values ​​in the correlation results as the visual relevance score for each word in the sentence.

4. A knowledge-oriented referential expression comprehension method according to claim 1, characterized in that: The step of calculating the correlation between the image visual features and the sentence text features and outputting the visual correlation score for each word in the sentence includes: Iterate T rounds to decompose the text feature vector x into multiple clauses and obtain the clause text feature vector x sub(T) , specifically expressed as: x sub(T) =s sub(T) ·x, s sub(T) =Conv(v sub(T-1) ·x·s sub(T-1) ) Where s sub(T-1) It is a score vector for calculating the clause position, Conv() is a convolution operation, and the clause visual feature vector of the Tth round is v sub(T) , specifically expressed as: in sub(T) =ReLU(in sub(T-1) ·Linear(x sub(T) )+Linear(x sub(T) )) In the formula, ReLU() is the activation function and Linear() is the linear function; Merge the clause text feature vector and clause visual feature vector obtained in round T to obtain the correlation x′ between the visual feature vector v and the text feature vector x sub , specifically expressed as: The correlation x′ sub Mapped to the visual relevance score x for each word in the sentence vi , specifically expressed as: x vi =Softmax(Linear(x′ sub ))·x+x。 5. The method for understanding referential expressions based on knowledge according to claim 1, wherein: The knowledge-related sentence fragments are input into the knowledge retrieval model, and the category name of the object described by the knowledge-related sentence fragments is output; The object category name is input into the text encoder, and the corresponding object category name text features are output, including: The knowledge-related sentence fragment S kn As a prompt word, input into the knowledge retrieval model E k Perform mask prediction in the text encoder E to obtain the category name of the object described by the sentence fragment related to the knowledge, and obtain the category name z of the top M most matching objects; input the category name z of the top M most matching objects into the text encoder E t In the output, the corresponding object category name text feature x z(M) .

6. A knowledge-oriented referential expression comprehension method according to claim 1, characterized in that: In the step of fusing the visually related sentence fragments with the text features of the object category name, the fusion operation includes item-by-item multiplication, item-by-item addition, feature connection, attention mechanism or a variant of the attention mechanism.

7. A knowledge-oriented referential expression comprehension method according to claim 1, characterized in that: The fusing operation of visually related sentence fragments with text features of object category names includes: The object category name text feature x z(M) Splice to sentence fragment S vi The first, last and middle parts of the concatenated vectors are then linearly mapped to the knowledge-related sentence features y M ; The fused features are input into the visual object localization model to output the predicted bounding box coordinates of the object, including: Iterate M rounds to transform the knowledge related sentence features y M Fused with the visual feature vector v to obtain the fused feature v kn , the fused feature v kn Input to the visual object localization model E g In the example, we get the predicted bounding box coordinates B of the object being referred to. pred .

8. The method for understanding referential expressions based on knowledge according to claim 1, wherein: The loss function of the visual object positioning model training is expressed as follows: in, is the error between the knowledge-related sentence fragment and the correct knowledge-related sentence fragment, is the smoothed L1 error between the predicted bounding box coordinates of the referent and the correct bounding box coordinates of the referent, is the GIoU error between the predicted bounding box coordinates of the reference object and the correct bounding box coordinates of the reference object, and λ1, λ2 and λ3 are adjustment factors.

9. A knowledge-oriented referential expression understanding device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 8 when executed by the processor.

Images