Text-image detection method based on multi-granularity decoder

Abstract

The present case relates to a text-image detection method based on a multi-granularity decoder, belonging to the field of image target detection, and is used to solve the problem that the current detection-related text-image task model cannot flexibly use language to describe objects in the image or must assume that the described object has and only has one. According to the task type, the present case configures a multi-granularity decoder with different granularities, obtains the fusion feature sequence of the image and the text in the task for a text-image task in which the target object in the image is described in the form of a character description and / or a bounding box, and outputs a text-image task detection result when the fusion feature sequence is taken as the input of the multi-granularity decoder. The present case unifies various text-image tasks related to detection and positioning into a text description object detection task, effectively unifying various task forms; the multi-granularity decoder has scalability, and the trained multi-granularity decoder can realize inference on different text-image tasks.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and general object detection, and specifically to a text-image detection method based on a multi-granularity decoder. Background Technology

[0002] General object detection has been an important research branch in computer vision for decades, covering images, videos, 3D data, and many other aspects. Related technologies are widely used in information retrieval, security monitoring, entertainment, and other fields. Among these, image-based detection, such as object detection and human-object interaction detection (HOI), has received relatively widespread attention. In recent years, with the continuous expansion of artificial intelligence applications, object detection tasks have gradually evolved towards open-vocabulary detection (OVD). While this task still involves detecting target objects from images, the objects being detected are no longer limited to categories defined by fixed vocabulary, but rather to a wider range of objects with arbitrary given category names, unrestricted by vocabulary. Another language-based object localization task is referring image comprehension, which locates the described object from an image based on a linguistic description of the image.

[0003] Open vocabulary detection tasks and pointing image understanding tasks share some similarities, both being based on language to locate objects in images. However, there are also significant differences: the former is a detection task where, for any category, there may be zero or more objects that need to be determined, while the latter simply locates the target from an image where the target is certain to exist; the target in the former is specified by the category name, while the latter uses a longer, more flexible, and more informative linguistic description.

[0004] Current work on these two tasks exhibits significant limitations: open-vocabulary detection methods, such as OWL-ViT and CORA, cannot flexibly use linguistic descriptions to specify the objects to be detected; while image understanding methods, such as OFA and PolyFormer, rely on the impractical assumption that the described object exists only once in the image. Some methods attempt to solve both tasks simultaneously using a multi-task approach, such as Grounding-DINO and UNINEXT, but these methods still retain separate training and inference methods for each task, thus preserving these existing problems. Summary of the Invention

[0005] To overcome the performance limitations of existing methods that are not applicable to object detection based on language descriptions, the present invention aims to provide a text-image detection method based on a multi-granularity decoder, which unifies various text-image tasks related to detection and localization into a text-description-based object detection task, effectively unifying various task formats.

[0006] To achieve the above objectives, the technical solution of this case is as follows.

[0007] Firstly, this case proposes a text-image detection method based on a multi-granularity decoder, including:

[0008] The method employs a multi-granularity decoder, which differs from the existing single encoder-decoder structure. The multi-granularity decoder is configured with decoders of different granularities according to the task type. For text-image tasks that describe target objects in images in the form of text descriptions and / or bounding boxes, the method obtains the fused feature sequence of images and text in the task.

[0009] When the fused feature sequence is used as input to the multi-granularity decoder, the multi-granularity decoder outputs the text-image task detection results.

[0010] In one embodiment of the above technical solution, the multi-granularity decoder includes a local granularity decoder and a global granularity decoder to resolve the granularity conflict between local target localization and global relationship understanding in text-image tasks. The local granularity encoder is configured to locate zero or more objects in the text description based on a fused feature sequence; the global granularity encoder is configured to determine whether the text description matches the located objects based on the fused feature sequence.

[0011] In the above technical solution, one way to obtain the fused feature sequence is to encode image features and text features by an encoder, thereby encoding text and image data in different text-object tasks in a unified form, so that different text-object tasks can be unified into a text-based object detection problem.

[0012] In the above technical solution, one implementation of text features is to obtain them by mapping the prompt and the text sequence using a word segmenter.

[0013] In the above technical solution, one training implementation of the multi-granularity decoder includes the following steps:

[0014] In one round of training, samples corresponding to different task types are mixed. At the end of training, the loss function values ​​of each granular decoder are calculated separately, and their sum is used as the overall loss.

[0015] In the above technical solution, the task includes a text description object detection task; the negative samples in the training samples of the text description object detection task are obtained by processing the positive samples, and the processing method includes: modifying the text description of the image target object in the positive samples to be incorrect, and translating the bounding box of the image target object in the positive samples.

[0016] In the above technical solution, when the multi-granularity decoder is applied to the task of object detection in text description, the steps include:

[0017] For any image, obtain a textual description of the target object in the image, and then obtain the first fusion feature sequence of the image and the textual description;

[0018] The multi-granularity decoder outputs the bounding box of the target object based on the first fused feature sequence;

[0019] The bounding box and the text description of the target object are concatenated, and a second fusion feature sequence is obtained based on the concatenated text and the image.

[0020] The multi-granular decoder determines whether the previously output bounding box of the target object matches the text description of the target object based on the second fused feature sequence.

[0021] In one embodiment of the above technical solution, the decoders of different granularities configured in the multi-granularity decoder configuration have the same structure.

[0022] Secondly, this case proposes a computer-readable storage medium storing a computer program that can be loaded by a processor and executed by any of the methods described above.

[0023] Thirdly, this case proposes a text-image detection system with a multi-granularity decoder, the system comprising an acquisition unit and a multi-granularity decoder; wherein:

[0024] The acquisition unit is configured to acquire the fused feature sequence of the image and text in a text-image task that describes a target object in an image in the form of a text description and / or a bounding box.

[0025] The multi-granularity decoder is configured with decoders of different granularities according to the task type; when the fused feature sequence is used as the input of the multi-granularity decoder, the multi-granularity decoder outputs the text image task detection result.

[0026] The beneficial technical effects of this case are:

[0027] (1) By encoding image features and text features into a unified fusion feature sequence, which serves as the unified input to a multi-granularity decoder, various text-image tasks related to detection and localization are unified into text-based object detection tasks, thus proposing a general processing model that effectively unifies the forms of various tasks. For tasks involving simultaneous detection and localization, the process is implemented step by step.

[0028] (2) The multi-granularity decoder is configured with decoders of different granularities according to the task type. It is scalable and can be applied to object detection for different text image tasks. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 , one A flowchart illustrating the workflow in one specific implementation method;

[0031] Figure 2 , one A schematic diagram of the voice-described object detection results in one specific implementation. Detailed Implementation

[0032] Currently, different text-image tasks require different task models, each trained and implementing its own task inference. Some text-image tasks are similar, such as open-vocabulary detection and pointing image understanding. The former is a detection task; for any category, objects may exist zero or more times, requiring judgment. The latter simply locates the target from an image where the target is certain to exist. The target in the former is specified by the category name, while the latter uses a longer, more flexible, and more informative linguistic description. This leads to significant limitations in related work on these two tasks: open-vocabulary detection methods, such as OWL-ViT and CORA, cannot flexibly use written language descriptions to specify the objects to be detected, while pointing image understanding methods, such as OFA and PolyFormer, rely on the impractical assumption that the described object exists exactly once in the image.

[0033] This paper proposes a text-based object detection method based on a multi-granularity decoder. The detection principle diagram is shown below. Figure 1As shown in the diagram, the method employed in this case utilizes a scalable multi-granularity decoder, which can be configured with decoders of different granularities according to the task type. Target objects in the image are described using text descriptions and / or bounding boxes. During text-described object detection, a fused feature sequence of the image and text in the task is obtained, and this fused feature sequence is used as input to the multi-granularity decoder. The multi-granularity decoder outputs the corresponding text-image task detection result according to the task type.

[0034] The following description, in conjunction with the accompanying drawings, clearly and completely describes how the technical solution of this case is implemented. Obviously, the described embodiments are only a part of the embodiments of this case, and not all of them. Based on the embodiments in this case, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0035] (I) Technical Solution

[0036] In one specific implementation, the method uses a general text-image model including multi-granularity decoders to unify different text-image tasks into a text-description-based object detection task, thereby effectively unifying the forms of various tasks. The multi-granularity decoders are configured with decoders of different granularities according to the task type, thus making the general text-image model scalable to be applicable to object detection and localization for different text-image tasks. In this case, the objects refer to all describable objects on the image. The model also includes an image backbone network based on residual convolutional networks, a word segmenter, and an encoder. The following describes each part of the model.

[0037] (1) Image backbone network

[0038] Image backbone networks are used to extract image features from text-image pairs. They take RGB images as input and output image features. Image backbone networks can be ResNet-50 or ResNet-101 networks pre-trained on the ImageNet dataset, ViT-B / ViT-L networks pre-trained on the ImageNet dataset, ResNet-50 networks in CLIP pre-trained models, ViT-B / ViT-L networks in CLIP pre-trained models, etc.

[0039] (2) Word segmenter

[0040] like Figure 2 As illustrated, the word segmenter takes text as input, maps the prompt to the text sequence to obtain text features, and outputs these text features. Text-image text is used to describe the target object in the image; it can be either text or the bounding box of the target object in the image.

[0041] (3) Encoder

[0042] The encoder is used to concatenate two sequences of text input features and image input features to form a fused feature sequence, thereby using data from different text-image tasks in a unified form as input to a multi-granularity decoder, so that different text-image tasks can be unified into a text-based object detection task.

[0043] (4) Multi-granularity decoder

[0044] The multi-granularity decoder in this case can be configured with decoders of different granularities according to the task type, and has scalability. The multi-granularity decoder unifies different task types as a text-based object detection problem.

[0045] In one round of training, samples corresponding to different task types are mixed. At the end of training, the loss function values ​​of each granular decoder are calculated separately, and their sum is used as the overall loss. The loss function used is the cross-entropy function.

[0046] In one implementation, the text-image task consists of a detection-localization subtask and a discrimination subtask. The detection-localization subtask includes, but is not limited to, standard object detection and pointing image understanding. The discrimination subtask includes image captioning, visual question answering, and mask language modeling. Based on the localization and discrimination subtasks, a local granular decoder and a global granular decoder are configured for the multi-granular encoder to achieve a unified multi-task model. Specifically: the local granular decoder is used to locate zero or more objects in the text description based on the fused feature sequence, outputting bounding boxes for zero or more target objects. The global granular decoder is used to determine whether the text description matches the located objects based on the fused feature sequence.

[0047] For existing localization and discrimination subtasks, such as open-vocabulary detection and pointing image understanding, after unifying them into a text-described object detection task, the trained multi-granular encoder can be implemented in two steps, including:

[0048] In the first inference, for any image, obtain a textual description of the target object in the image, and then obtain the first fused feature sequence of the image and the textual description;

[0049] The multi-granularity decoder is based on the first fused feature sequence and uses a local granularity encoder to output the bounding box of the target object.

[0050] The second reasoning involves concatenating the bounding box and the text description of the target object, and obtaining a second fusion feature sequence based on the concatenated text and the image.

[0051] The multi-granularity decoder, based on the second fused feature sequence, uses a global granular encoder to determine whether the previously output bounding box of the target object matches the text description of the target object. For example, a visual question-answering task is adopted, outputting a predefined result. For instance, if the result is "yes," the text description matches the bounding box of the target object, and this bounding box is used as the final output of the text description object detection. If the result is "no," the text description does not match the bounding box of the target object, and the text description object detection task does not output any detection result for that image. Alternatively, the word "yes" or "no" can be directly output to indicate whether the text description matches the object's bounding box.

[0052] In one implementation, the local granular decoder and the global granular decoder have the same structure; for example, they both employ a transformer decoder structure.

[0053] For a multi-granularity decoder consisting of local and global granular decoders, samples from the local and global granular decoders are mixed during a single training round. During training, based on the task type of each sample, only the output of one of the global or local granular decoders is supervised, while the output of the other is not supervised. At the end of training, the loss function values ​​of the local and global granular decoders are calculated separately, and their sum is used as the overall loss. Compared to existing single encoder-decoder structures, multi-granularity decoders better resolve the granularity conflict between local object localization and global relation understanding in tasks such as open vocabulary detection and pointing image understanding.

[0054] For the training samples of the text-described object detection task, in the sample dataset of the global granular decoder, positive samples consist of a sample image, the text description of the sample, and the corresponding bounding box of the target object, and the text description and the bounding box of the target object are matched. Negative samples come from two sources: one is composed of the image of each sample in the dataset, the bounding box corresponding to the text description of the sample, and the text descriptions of other samples, i.e., negative samples with incorrect text descriptions; the other is composed of the image of each sample in the dataset, the text description of the sample, and the bounding box of the target object after translation, i.e., negative samples with incorrect target bounding boxes.

[0055] In the above embodiments, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0056] (II) Experimental Comparison

[0057] To verify the actual performance of this case on the text-based object detection task, it was tested on three well-known public datasets for image understanding: RefCOCO, RefCOCO+, and RefCOCOg, and a written language description object detection dataset D. 3 Experiments were conducted on the RefCOCO, RefCOCO+, and RefCOCOg datasets. All three datasets used image and object bounding boxes from the COCO dataset, with accuracy as the evaluation metric. RefCOCO contains 142,209 written expressions from 19,994 images representing 50,000 objects. RefCOCO+ contains 141,564 written expressions from 19,992 images representing 49,856 objects. Compared to RefCOCO, RefCOCO+ lacks location words in its written expressions, making the task more challenging. RefCOCOg contains 85,474 object-pointing expressions from 26,711 images representing 54,822 objects. These written expressions were collected from Amazon Mechanical Turk, resulting in longer and more complex descriptions. The average length of written expressions in RefCOCOg is 8.4 words, compared to 3.5 words in RefCOCO and RefCOCO+. The test data for RefCOCO and RefCOCO+ are divided into three parts: val, testA, and testB, while RefCOCOg is divided into two parts: val and test. 3 This dataset is a newly proposed written language description object detection dataset. Compared to previous pointing image understanding datasets, all written language descriptions are labeled with positive and negative samples across the entire dataset rather than on individual images. Therefore, this dataset is applicable to written language description detection tasks, while traditional pointing image understanding datasets are not. The images are sourced from websites such as Flickr and are manually annotated. Only a test set is provided, containing 10,578 images and 18,514 bounding boxes. Due to its complete annotation, the dataset contains a very large number of 16,480 positive sample instances and 5,833,944 negative sample instances. The dataset is also evaluated using accuracy as the metric, with three evaluation settings: FULL, PRES, and ABS.

[0058] Table 1 shows a comparison of the performance of the present invention with existing methods on the test set of the Pointing Image Understanding Dataset RefCOCO. The present invention outperforms the best existing methods on the RefCOCO dataset.

[0059] Table 1

[0060]

[0061] Table 2 presents a comparison of accuracy metrics between the present invention and other methods that perform well on the RefCOCO+ dataset for pointing image understanding. Similarly, the present invention surpasses the best existing technology on the RefCOCO+ dataset.

[0062] Table 2

[0063]

[0064]

[0065] Similar to Tables 1 and 2, Table 3 presents a comparison of the accuracy metrics of this invention with other state-of-the-art methods on the RefCOCOg dataset. The combined results from Tables 1, 2, and 3 show that although the proposed method is designed for text-described object detection tasks, it also achieves a leading level in pointing image understanding tasks.

[0066] Table 3

[0067]

[0068] Table 4 shows the performance of the proposed method and previous methods on the language description object detection dataset D. 3 The comparisons are as follows. The methods used for comparison are derived from traditional pointing image understanding (OFA) or open vocabulary detection tasks (CORA, OWL-ViT), or methods that combine both tasks (Grounding DINO, UNINEXT). As can be seen from Table 4, the method proposed in this invention has significant advantages over previous methods in the language description object detection task.

[0069] Table 4

[0070]

[0071]

[0072] As can be seen from Tables 1 to 4, this method achieved overall leadership across two tasks, four datasets, and a total of 11 metrics, reaching the highest level in the vast majority of metrics.

[0073] Figure 2The results of this invention, compared with previous pointing image understanding methods (OFA), open vocabulary detection methods (OWL-ViT), and methods that combine both (Grounding-DINO), can simultaneously (1) avoid duplicate and excessive detection results, (2) not make predictions on images without target instances, and the prediction results are closer to the real annotations, demonstrating the excellent effect of the method of this invention in language description object detection.

[0074] In summary, the proposed method performs better on generalized language-based object detection tasks. It is not only stronger than previous methods on traditional pointing image understanding tasks, but also more suitable for the latest text-based object detection tasks. The accuracy of the results is significantly improved on various types of data, and it has certain practical value.

[0075] In another embodiment, a text description object detection system with a multi-granularity decoder is implemented according to the above method. The system includes an acquisition unit and a multi-granularity decoder; wherein:

[0076] The acquisition unit is configured to acquire the fused feature sequence of the image and text in a text-image task that describes a target object in an image in the form of a text description and / or a bounding box.

[0077] The multi-granularity decoder is configured with decoders of different granularities according to the task type; when the fused feature sequence is used as the input of the multi-granularity decoder, the multi-granularity decoder outputs the text-image task detection result.

[0078] If the above-described methods or systems are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0079] It should be noted that the terms "one embodiment," "another embodiment," and "embodiment" used in this specification refer to specific features, structures, or characteristics described in connection with that embodiment, which are included in at least one embodiment described in the general description of this application. The appearance of the same expression in multiple places in the specification does not necessarily refer to the same embodiment. Furthermore, when a specific feature, structure, or characteristic is described in connection with any embodiment, the intention is to suggest that implementing such a feature, structure, or characteristic in conjunction with other embodiments also falls within the scope of this invention.

Claims

1. A text-image detection method based on a multi-granularity decoder, characterized in that: The method described herein is used to complete the text-described object detection task. It configures decoders of different granularities according to the task type to form a multi-granularity decoder, including: dividing the task type into localization subtasks and discrimination subtasks according to the task objective, and configuring local granular decoders and global granular decoders respectively to form a multi-granularity decoder; during training, mixing samples corresponding to the local granular decoder and samples corresponding to the global granular decoder; and supervising only the output of one of the global granular decoder and the local granular decoder according to the task type of each sample, while not supervising the output of the other; at the end of training, calculating the loss function values ​​of the local granular decoder and the global granular decoder respectively, and using the sum of the two as the overall loss. For text-image tasks that describe target objects in an image using text descriptions and / or bounding boxes, where the text-image task is a text-based object detection task, the steps include: obtaining a fused feature sequence by concatenating the image features of the input image and the text features of the input text; and when the fused feature sequence is used as input to a multi-granularity decoder, the multi-granularity decoder outputs the text-image task detection result. In the localization subtask, the local granular decoder locates zero or more objects in the text description based on the fused feature sequence, and outputs bounding boxes for zero or more target objects; When there is a discriminative subtask, the bounding box and the text description of the target object are further concatenated. Based on the fused feature sequence obtained after concatenation, the global granular decoder of the multi-granular decoder determines whether the previously output bounding box of the target object matches the text description of the target object.

2. The method according to claim 1, characterized in that, Text features are obtained by mapping the prompt to the text sequence using a word segmenter.

3. The method according to claim 1, characterized in that: Negative samples in the training samples for the text description object detection task are obtained by processing positive samples. The processing methods include: for a positive sample, modifying its text description of the target object in the image to be incorrect or translating the bounding box of the target object in the image.

4. The method according to claim 1, characterized in that, Decoders of different granularities have the same structure.

5. A computer-readable storage medium, characterized in that: The computer program is stored that can be loaded by a processor and executed according to any one of claims 1 to 4.

6. A text-image detection system with a multi-granularity decoder implemented according to claim 1, characterized in that, The system includes an acquisition unit and a multi-granularity decoder; wherein: The acquisition unit is configured to acquire the fused feature sequence of the image and text in a text-image task that describes a target object in an image in the form of a text description and / or a bounding box. The multi-granularity decoder is configured with decoders of different granularities according to the task type; when the fused feature sequence is used as the input of the multi-granularity decoder, the multi-granularity decoder outputs the text-image task detection result.

Images