A scene text-aware referential expression understanding method, device and storage medium

Abstract

The present invention discloses a scene text-aware referential expression understanding method, device and storage medium, wherein the method comprises: explicitly extracting scene text instances in an image, calculating the correlation between the scene text instances and referential expression sentences, and then calculating the spatial position of the scene text instances in the image features to activate the relevant areas of the scene text instances in the image features, and finally refining the scene text-aware image features and sentence text features to predict the position coordinates of the referential target object. The present invention solves the problem that the current referential expression understanding technology cannot explicitly align scene text information, resulting in incorrect positioning of the target object, and can improve the robot's cognitive ability for scene text, visual objects and visual scenes, and improve the human-computer interaction ability for scene text. The present invention can be widely used in the fields of natural language processing and computer vision technology.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and computer vision technology, and in particular to a scene text-aware referential expression understanding method, device and storage medium. Background Art

[0002] Referential expression understanding refers to identifying and locating the object indicated by the referential expression sentence in the visual scene. As a research branch of multimodal machine learning, this task requires both analyzing the visual scene and understanding the text semantics, and then integrating and aligning the information of the visual modality and the language modality. From the application level, referential expression understanding is the basic work of multimodal machine learning, providing multimodal object retrieval capabilities for downstream applications such as visual intelligent question answering and visual navigation. In terms of engineering value, natural language is the most direct way of communication for human-computer interaction and human-computer collaboration. The referential expression understanding task is a basic research work that enables machines to have the intelligence to autonomously understand referential intentions.

[0003] The text that can be seen everywhere in visual scenes is called scene text, which carries rich semantic connotations. Scene text gives objects identification and descriptive information and is an important medium for information exchange between people and objects. Because scene text has the characteristics of identification and differentiation, humans usually use text on the surface or around objects to refer to objects.

[0004] However, existing referential expression understanding technologies lack the recognition and processing of scene text. The general visual encoders they use cannot extract text from images, making it difficult for the machine to understand the scene text mentioned in referential expression sentences, and thus mistakenly aligning the mentioned scene text to irrelevant visual areas or visual features, resulting in incorrect positioning of the target object. Summary of the invention

[0005] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a scene text-aware referential expression understanding method, device and storage medium.

[0006] The technical solution adopted by the present invention is:

[0007] A scene text-aware referential expression understanding method comprises the following steps:

[0008] Obtain a referential expression understanding dataset for scene text, where each sample in the referential expression understanding dataset consists of an image, a sentence, and the coordinates of a bounding box of a correct referent object;

[0009] Input the sentence of each sample into the text encoder and output the corresponding sentence text features; input the image of each sample into the visual encoder and output the corresponding image features; input the image of each sample into the scene text recognition and detection model and output the text and position coordinates of the corresponding scene text instance;

[0010] Calculate the similarity between the text of the scene text instance in the image and the words in the sentence, and if the similarity is less than a first preset threshold, replace the text of the scene text instance with the words in the sentence; input the text of the scene text instance into a text encoder, and output the text features of the corresponding scene text instance;

[0011] Calculate the correlation between the text features of the scene text instance in the image and the text features of the words in the sentence to obtain a correlation score of the scene text, and if the correlation score is less than a second preset threshold, remove the scene text instance;

[0012] Construct a visual feature map, calculate the spatial position of the scene text instance on the visual feature map according to the position coordinates of the scene text instance, and assign a relevance score of the scene text at the corresponding spatial position;

[0013] The visual feature map is fused with the image features to obtain the image features perceived by the scene text;

[0014] Obtain a vision-and-language alignment framework, input the scene-text-aware image features and sentence text features into the vision-and-language alignment framework, and output the bounding box coordinates of the corresponding referent object.

[0015] Furthermore, the text encoder includes RNN, RNN variants, and BERT series; wherein the RNN variants include: LSTM series, GRU series;

[0016] The visual encoder includes ResNet, DarkNet, and DETR;

[0017] The scene text recognition and detection models include Google OCR, PaddlePaddle OCR, Baidu OCR, and Rosetta.

[0018] Furthermore, in the step of calculating the similarity between the text of the scene text instance in the image and the words in the sentence, the calculation methods adopted include edit distance, Euclidean distance, cosine distance, and Jaccard distance.

[0019] Furthermore, in the step of calculating the correlation between the text features of the scene text instance in the image and the text features of the words in the sentence, the calculation method used includes edit distance, Euclidean distance, cosine distance, and Jaccard distance;

[0020] The relevance score of the scene text is obtained, including:

[0021] Get the maximum value, average value, or median value of each row or column in the relevance result as the relevance score of the scene text.

[0022] Further, in the step of calculating the spatial position of the scene text instance on the visual feature map according to the position coordinates of the scene text instance, the spatial position is represented by a grid, and the left boundary, the upper boundary, the right boundary, and the lower boundary of the grid are specifically obtained by dividing the position coordinates of the scene text instance by the scaling scale of the image feature dimension;

[0023] The assigning a relevance score of the scene text at the corresponding spatial position includes:

[0024] If the grid falls on the corresponding position of the image feature, the relevance score of the scene text is assigned to this position, and the remaining positions without scene text are assigned default values.

[0025] Furthermore, in the step of fusing the visual feature map with the image features, the fusion operation includes item-by-item multiplication, item-by-item addition, and feature connection.

[0026] Furthermore, the vision and language alignment framework includes ReSC, LBYL, and BBA.

[0027] Furthermore, the loss function of the vision and language alignment framework training is expressed as:

[0028] L=λ1L c +L s +λ2L align

[0029] Among them, K c is the error between the bounding box coordinates output by the model and the bounding box coordinates that correctly refer to the object, L s L is the size offset between the bounding box coordinates output by the model and the bounding box coordinates that correctly refer to the object. align is the error of the intermediate variable in the vision and language alignment framework; λ1 and λ2 are adjustment factors used to balance the loss function.

[0030] Another technical solution adopted by the present invention is:

[0031] A scene text-aware referential expression understanding device, comprising:

[0032] at least one processor;

[0033] at least one memory for storing at least one program;

[0034] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0035] Another technical solution adopted by the present invention is:

[0036] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to execute the method described above when executed by the processor.

[0037] The beneficial effects of the present invention are as follows: the present invention utilizes a scene text recognition and detection model to explicitly extract scene text from an image and encode its spatial position information and semantic information, and aligns the scene text with sentences and images respectively, thereby solving the problem that current referential expression understanding technology cannot explicitly align scene text information, resulting in incorrect positioning of target objects. The present invention can enhance the robot's cognitive ability of scene text, visual objects, and visual scenes, and enhance the human-computer interaction capability for scene text. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 It is a flow chart of a method for scene text-aware referential expression understanding in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0041] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0042] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0043] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention in combination with the specific content of the technical solution.

[0044] like Figure 1 As shown, this embodiment provides a scene text-aware referential expression understanding method, and the specific steps include:

[0045] S1. Obtain a dataset for scene text-oriented referential expression understanding. Each sample in the dataset consists of an image I and a sentence containing N words. The text encoder TE is obtained, which is used to extract the text features of the sentence. The visual encoder VE is obtained, which is used to extract the visual features of the image. The scene text recognition and detection model STE is obtained, which is used to extract the text and position coordinates of the scene text instance in the image.

[0046] S2. Input each sample sentence S into the text encoder TE, and the model outputs the corresponding sentence text features The text features contain N words The image I of each sample is input into the visual encoder VE, and the model outputs the corresponding image feature F i ; Input the image of each sample into the scene text recognition and detection model STE, and the model outputs the text of the corresponding M scene text instances With position coordinates

[0047] In an optional embodiment, the text encoder TE is BERT, which encodes the input sentence S into a sentence text feature F containing N=20 words. s , the text of each scene text instance Text features encoded as scene text containing N=20 words

[0048] In an optional embodiment, the visual encoder VE is DarkNet, which encodes the input image I into image features F i, the image feature contains L = 3 layers of features, each layer of feature dimension is d l .

[0049] S3. Calculate the text of each scene text instance in the image With each word s in the sentence n SimilaritySim m,n , specifically expressed as:

[0050]

[0051] Where Dis is the edit distance and ‖·‖ is the string length.

[0052] If the similarity Sim m,n is less than the threshold θ, then use the word s in the sentence n Replace the text of a scene text instance Afterwards, the text Input into the text encoder TE, the model outputs the text features of the corresponding scene text

[0053] S4. Calculate the text features of each scene text instance in the image The text features of each word in the sentence The correlation of A m,n , specifically expressed as:

[0054]

[0055] Take A m,n The maximum value of each row in is taken as the relevance score R of the scene text m , specifically expressed as:

[0056]

[0057] If the correlation score R m Less than threshold Then set the score to 0.

[0058] S5. Construct a visual feature map E with all values ​​​​0, based on the position coordinates of the scene text instance Calculate the position of the mth scene text instance E on the visual feature map, which is represented by a grid with a left boundary b lt , upper boundary b up , right boundary b rt , lower boundary b low Specifically expressed as:

[0059]

[0060]

[0061]

[0062]

[0063] Among them, W and H are image features F i The width and height, d l is the image feature F i The l-th layer feature dimension, the location coordinates of the scene text instance are

[0064] Assign the relevance score R of the scene text at the corresponding position m , specifically expressed as:

[0065]

[0066] S6. Fusing the visual feature map with the image features to obtain scene text-aware image features.

[0067] In an optional embodiment, the visual feature map E and the image feature F i Perform a multiplication operation item by item, and the result is the same as the image feature F i Perform item-by-item addition operations to obtain the scene text-aware image feature F′ i .

[0068] S7. Obtain a vision and language alignment framework, input the scene text-perceived image features and sentence text features into the vision and language alignment framework, and output the corresponding bounding box coordinates of the reference object.

[0069] The scene text-aware image feature F′ i and sentence text feature F s Map them to dimension C respectively; obtain the vision and language alignment framework Z, and transform the scene text-perceived image features F′ i and sentence text feature F s Input into the vision and language alignment framework Z, the model outputs the corresponding bounding box coordinates P of the object.

[0070] In an optional embodiment, the vision and language alignment framework Z is a BBA model, and the loss function of the model training is expressed as:

[0071] L=λ1L c +L s +λ2L align

[0072] Among them, L c is the error between the bounding box coordinates P output by the model and the bounding box coordinates B that correctly refers to the object, L sL is the size offset between the bounding box coordinates P output by the model and the bounding box coordinates B that correctly refers to the object. align is the error of the intermediate variable of the BBA model; λ1 and λ2 are adjustment factors used to balance the loss function.

[0073] This embodiment also provides a scene text-aware referential expression understanding system, including: a feature encoding module, a related text extraction module, a related region activation module, and a refinement and prediction module;

[0074] The feature encoding module is used to extract sentence text features, image features, text features and position coordinates of scene text instances;

[0075] The related text extraction module is used to calculate the correlation between the sentence text features and the text features of the scene text instance, and obtain the correlation score R of the scene text instance. m ;

[0076] The related region activation module is used to convert the correlation score R m Assign to the location of the scene text instance on the visual feature map E, and combine the visual feature map E with the image feature F i After fusion, we get the scene text-aware image feature F′ i ;

[0077] The refinement and prediction module is used to transform the scene text-aware image features F′ i and sentence text feature F s Input into the vision and language alignment framework Z, after being refined in Z, the corresponding bounding box coordinates P of the referring object are output.

[0078] A scene text-aware referential expression understanding system of this embodiment can execute a scene text-aware referential expression understanding method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0079] This embodiment also provides a scene text-aware referential expression understanding device, including:

[0080] at least one processor;

[0081] at least one memory for storing at least one program;

[0082] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 The method shown.

[0083] A scene text-aware referential expression understanding device of this embodiment can execute a scene text-aware referential expression understanding method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0084] The present application also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and a scene text-aware denotative expression understands and executes the computer instruction, so that the computer device executes Figure 1 The method shown.

[0085] This embodiment also provides a storage medium storing instructions or programs that can execute a scene text-aware referential expression understanding method provided by the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0086] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0087] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0088] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0089] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0090] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0091] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0092] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0093] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0094] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A scene text-aware referential expression understanding method, characterized in that: The following steps are involved: Obtain a referential expression understanding dataset for scene text, where each sample in the referential expression understanding dataset consists of an image, a sentence, and the coordinates of a bounding box of a correct referent object; Input each sample sentence into the text encoder and output the corresponding sentence text features; Input the image of each sample into the visual encoder and output the corresponding image features; Input the image of each sample into the scene text recognition and detection model, and output the text and position coordinates of the corresponding scene text instance; Calculate the similarity between the text of the scene text instance in the image and the words in the sentence, and if the similarity is less than a first preset threshold, replace the text of the scene text instance with the words in the sentence; input the text of the scene text instance into a text encoder, and output the text features of the corresponding scene text instance; Calculate the correlation between the text features of the scene text instance in the image and the text features of the words in the sentence to obtain a correlation score of the scene text, and if the correlation score is less than a second preset threshold, remove the scene text instance; Construct a visual feature map, calculate the spatial position of the scene text instance on the visual feature map according to the position coordinates of the scene text instance, and assign a relevance score of the scene text at the corresponding spatial position; The visual feature map is fused with the image features to obtain the image features perceived by the scene text; Obtain a vision-language alignment framework, input the scene text-aware image features and sentence text features into the vision-language alignment framework, and output the bounding box coordinates of the corresponding reference object.

2. A scene text-aware referential expression understanding method according to claim 1, characterized in that: The text encoder includes RNN, RNN variants, and BERT series; The visual encoder includes ResNet, DarkNet, and DETR; The scene text recognition and detection models include Google OCR, PaddlePaddle OCR, Baidu OCR, and Rosetta.

3. The scene text-aware referential expression understanding method according to claim 1, characterized in that: In the step of calculating the similarity between the text of the scene text instance in the image and the words in the sentence, the calculation methods used include edit distance, Euclidean distance, cosine distance, and Jaccard distance.

4. The scene text-aware referential expression understanding method according to claim 1, characterized in that: In the step of calculating the correlation between the text features of the scene text instance in the image and the text features of the words in the sentence, the calculation methods used include edit distance, Euclidean distance, cosine distance, and Jaccard distance; The step of obtaining the relevance score of the scene text includes: Get the maximum value, average value, or median value of each row or column in the relevance result as the relevance score of the scene text.

5. The scene text-aware referential expression understanding method according to claim 1, characterized in that: In the step of calculating the spatial position of the scene text instance on the visual feature map according to the position coordinates of the scene text instance, the spatial position is represented by a grid, and the left boundary, the upper boundary, the right boundary, and the lower boundary of the grid are specifically obtained by dividing the position coordinates of the scene text instance by the scaling scale of the image feature dimension; The assigning a relevance score of the scene text at the corresponding spatial position includes: If the grid falls on the corresponding position of the image feature, the relevance score of the scene text is assigned to this position, and the remaining positions without scene text are assigned default values.

6. The scene text-aware referential expression understanding method according to claim 1, characterized in that: In the step of fusing the visual feature map with the image features, the fusion operation includes item-by-item multiplication, item-by-item addition, and feature connection.

7. The method for scene text-aware referential expression understanding according to claim 1, characterized in that: The vision and language alignment framework includes ReSC, LBYL, and BBA.

8. The scene text-aware referential expression understanding method according to claim 1, characterized in that: The loss function of the vision and language alignment framework training is expressed as: L=λ1L c +L s +λ2L align Among them, L c is the error between the bounding box coordinates output by the model and the bounding box coordinates that correctly refer to the object, L s L is the size offset between the bounding box coordinates output by the model and the bounding box coordinates that correctly refer to the object. align is the error of the intermediate variable in the vision and language alignment framework; λ1 and λ2 are adjustment factors used to balance the loss function.

9. A scene text-aware referential expression understanding device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 8 when executed by the processor.

Images