General visual relationship recognition detection method based on multi-modal large model

Abstract

The present case relates to a general visual relationship recognition detection method based on a multi-modal large model, belongs to the field of computer vision and visual relationship recognition, and is used to solve the problem that the expert models of each sub-task lack generality, flexibility and language understanding ability in the current visual relationship recognition detection task. After the multi-modal large model is trained by the constructed data samples, it can be applied to different visual relationship recognition detection tasks and different data sets without special architecture adjustment or fine tuning for a certain task or data set. Moreover, the multi-modal large model has certain dialogue ability, can directly and effectively utilize the language modal information of the large model and its language and knowledge reasoning ability, adjust the task purpose and output form according to the user's answer, and can flexibly compatible with more task forms such as visual relationship reference.

Description

Technical Field

[0001] This invention relates to computer vision and visual relationship recognition, and more particularly to a general visual relationship recognition and detection method based on a multimodal large model. Background Technology

[0002] Visual relationships are crucial for visual perception and reasoning, encompassing tasks such as Human-Object Interaction Detection (HOI), Scene Graph Generation (SGG), and Task-Driven Object Detection (TDOD).

[0003] Despite significant efforts, this field still faces several limitations: First, while there are connections between visual relational tasks, most previous work has focused on building an expert for a single task, using heuristics and task-specific modules rather than a unified model. Second, different tasks have specific, rigorous, and complex formulas, such as structural triads detected in HOI and SGG, which limits flexibility for various practical needs. Finally, visual relational tasks naturally require reasoning and knowledge to understand important factors such as body language, object function, common sense, and context. Recent research, while employing language models and knowledge, has focused on using coarsely aligned models to provide additional input or supervision, rather than fully leveraging reasoning based on language and knowledge. Summary of the Invention

[0004] The purpose of this study is to propose a general visual relationship recognition and detection method and system based on a multimodal large model, in order to solve the problem that the expert models for various sub-tasks in the current visual relationship recognition and detection tasks lack universality, flexibility and language understanding ability.

[0005] To achieve the aforementioned technical objectives, the technical solution in this case is as follows.

[0006] Firstly, this case proposes a general visual relationship recognition and detection method based on a multimodal large model. The steps include: obtaining task elements based on real visual relationships in the task, wherein the task elements include relationship category, object position, object category, whether there is interaction, and object function; reconstructing visual relationships based on task elements to obtain pseudo data samples; and training the multimodal large model using real data samples and pseudo data samples, wherein the trained multimodal large model can perform general visual relationship recognition and detection.

[0007] As one implementation of the above technical solution, the steps of reconstructing visual relationships based on task elements include: reconstructing visual relationships using thought chains based on task element chains; or outputting task elements according to one or more rounds of dialogue prompts based on task element chains, obtaining a reconstructed visual relationship in each round; or using a link between the input node and the output element node of the task element chain as a reconstructed visual relationship; or obtaining the dependency relationships of task elements in the real visual relationship, obtaining indirect dependency relationships based on the dependency relationships, and using the indirect dependency relationships to combine task elements to reconstruct visual relationships.

[0008] As one implementation of the above technical solution, the steps of reconstructing visual relationships based on task elements include: for real visual relationships, if the acquired task elements include object position, object category, whether there is interaction, object function, and relationship category, setting a confidence level for each object corresponding to the object position, object category, and object function, and setting an associated confidence level for two objects involved in whether there is interaction or relationship category; adjusting the detection box to adjust the position of the object, or to change the label of the object category or object function, or to pair and adjust two objects involved in whether there is interaction or relationship category, or to randomly adjust using object detection annotations in the dataset, calculating and updating the corresponding confidence level, thereby obtaining new visual relationships.

[0009] As one implementation of the above technical solution, if there are multiple real visual relationships in the task, an additional placeholder is added after each real relationship category label; if the task elements of the real visual relationships in the task include object functions, an additional placeholder is added after their corresponding labels; wherein, the placeholders are not used to calculate loss during training.

[0010] As one implementation of the above technical solution, the step of reconstructing visual relationships based on task elements includes: providing a selection range of category names in the dialogue prompt.

[0011] As one implementation of the above technical solution, the step of reconstructing visual relationships based on task elements includes: reconstructing new visual relationships using the labels corresponding to all visual relationships, thereby establishing a correspondence between a portion of task elements and another portion of task elements.

[0012] As one implementation of the above technical solution, when the multimodal large model performs visual relationship recognition and detection, it takes (I, T) as input, where I is an image and T is a text sequence, and outputs the recognized visual relationship in the form of a text response; wherein: M represents the number of conversations. and These are the text questions and text answers for the m-th dialogue.

[0013] As one implementation of the above technical solution, the multimodal large model uses two image backbone networks of different sizes to extract image features. The output layer of the larger image backbone network is followed by a pooling layer to reduce the length of the corresponding features. The obtained image features are each processed by a multilayer perceptron to obtain two image feature sequences. A text feature sequence is obtained based on the text sequence, and it is merged with the image feature sequence into an image-text feature sequence and input into the large language model. For each position of the image-text feature sequence, the large language model predicts the next word based on all the preceding words.

[0014] As one implementation of the above technical solution, the text feature sequence is obtained by processing the text sequence through a word segmenter.

[0015] Secondly, this case proposes a computer-readable storage medium that can store a computer program that can be loaded by a processor and executed by any of the methods described above.

[0016] The beneficial technical effects of this case are as follows: (1) Compared with the previous model architectures that were specifically designed for each visual relationship task and were only applicable to a single task and a single dataset, the multimodal large model trained in this case can be uniformly applied to different visual relationship recognition and detection tasks and different datasets without the need for special architecture adjustments or fine-tuning for a specific task or dataset. (2) The general visual relationship recognition and detection model in this case has a certain dialogue capability, directly and effectively utilizing the language modality information of the large model and its language and knowledge reasoning capabilities. It can adjust the task purpose and output form according to the user's answer and can flexibly accommodate more task forms such as visual relationship reference. (3) This case proposes a method for constructing pseudo-samples based on real visual relationships. This method does not require the addition of new images and text, can meet the training requirements of multimodal large models, and helps multimodal large models improve their ability to recognize and detect various visual relationships. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 , one A schematic diagram of the model architecture and input / output format in a specific implementation method.

[0019] Figure 2 , one A flowchart illustrating the workflow in one specific implementation method.

[0020] Figure 3 , one A schematic diagram of task splitting and reorganization based on mind chain in a specific implementation.

[0021] Figure 4 , one A schematic diagram illustrating the construction of a pseudo-instance containing confidence labels in one specific implementation. Detailed Implementation

[0022] Most existing research typically focuses on only one of the many visual relationship recognition and detection tasks, without considering their similarities or potential for mutual assistance. These studies often concentrate on a fixed task prediction format, neglecting the flexibility of other formats more suitable for various application scenarios. A few works, such as UniVRD, have addressed the similarities between these tasks and attempted to fuse two of them (human-object interaction detection and scene graph generation), but lack the ability to handle more task formats. Recently, some methods have begun to explore the application potential of large language models or multimodal large models in visual relationship-related tasks. UniHOI, for example, applies this to the single task of human-object interaction detection, improving performance on the traditional human-object interaction detector architecture, but it does not fuse different tasks or provide flexibility for more task formats. Currently, there are no methods directly based on multimodal large models that can handle multiple visual relationship tasks.

[0023] Based on this, this paper proposes a general visual relationship recognition and detection method based on a multimodal large model. In this method, the multimodal large model is trained using data samples designed and constructed in this paper. The resulting multimodal large model can perform visual relationship recognition and detection for a variety of tasks without the need for special architectural adjustments or fine-tuning for a specific task or dataset.

[0024] Specifically, the method obtains task elements based on real visual relationships in the task. These task elements include relationship category (R), object position (O), object category (C), presence or absence of interaction (B), and object function (A). Visual relationships are reconstructed based on these task elements to obtain pseudo-data samples. A multimodal large-scale model is trained using both real and pseudo-data samples. The trained multimodal large-scale model is capable of general visual relationship recognition and detection. Visual relationship tasks mainly include human-object interaction detection, scene graph generation, and task-driven object detection.

[0025] The following will be combined with the appendix Figure 1-4This application provides a clear and complete description of how the technical solution in this case is implemented. Obviously, the described implementation methods are only a part of the implementation methods in this case, not all of them. Based on the implementation methods in this case, all other implementation methods obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0026] (I) Unified Model

[0027] See Figure 1 This illustration shows a model framework and the supported visual relationship tasks in one implementation. Figure 1 The model in the model is a multimodal large model. When performing visual relationship recognition and detection, it takes (I, T) as input, where I is the image and T is the text sequence, and outputs the recognized visual relationship in the form of a text response. M represents the number of conversations. and These are the text questions and text answers for the m-th dialogue.

[0028] Image I is input into the visual encoder, and image features are obtained through feature projection. Text features are then obtained based on the text. The image features and text features are concatenated and used as input to the large language model. During model training, the input and output text features are the text question and text answer features. During the testing or application phase after model training, the input text features are the text question features, and the output text features are the text answer features.

[0029] In one implementation, the visual encoder is composed of two image backbone networks of different sizes, see [link to implementation details]. Figure 2 One network is a ViT-L image encoder network with an input size of 336 (initialized by CLIP), and the other is a ViT-L image encoder network with an input size of 1024 (initialized by SAM). Therefore, the image is processed into these two sizes before input. These two sizes enable general recognition and detection of visual relationships and improve the accuracy of recognition and detection. It is worth noting that because the large-size image encoder network generates a large number of image features, it uses pooling layers to process the obtained image features to reduce the length of this part of the image feature sequence. For example, the number of pooling layers is 1.

[0030] In one implementation, the feature projection described above employs a multilayer perceptron architecture. More specifically, it consists of two parallel multilayer perceptrons. These two perceptrons respectively receive image feature sequences output from two image backbone networks and map them into the same feature space as the text features output from the word segmenter. This allows image features and text features to be used as unified input to the large language model, and the task of predicting the next word is performed on the output feature sequence of the large language model.

[0031] In one implementation, text features are obtained by mapping a text sequence composed of multiple rounds of dialogue using a word segmenter.

[0032] In one implementation, the large language model employs a transformer architecture. Specifically, it is a transformer decoder, which can use 11ama as the large language model. This large language model receives an image-text feature sequence as input and, at each position in the sequence, predicts the next word based on all preceding tokens (corresponding to each word in the feature sequence) (see [link to implementation]). Figure 1 At the end of training, the loss function value for each position in the sequence is calculated separately, and their average is used as the overall loss. The loss function used is the cross entropy function.

[0033] This multimodal large model possesses a certain level of dialogue capability. Leveraging the flexibility of language prompts, the model can effectively utilize the language modality information and its inherent language and knowledge reasoning abilities. Furthermore, it can adjust the task objective and output format based on user responses, and is flexible enough to accommodate various task formats such as visual relational reference. In one implementation, within the text sequence, the dialogue question contains the task prompt and some input information required for the task; the dialogue response contains the task's prediction target, such as category labels or object bounding boxes. An example is shown below:

[0034] Question: For what purpose should the wine be supplied?

[0035] Answer: Wine glass [xyxy].

[0036] The [xyxy] format described above represents one type of object detection bounding box format, where xyxy indicates the use of top-left and bottom-right coordinates. The text input (question) includes task cues (such as the object to be used above), and sometimes also includes sampled input labels (such as the wine mentioned above). The text output (answer) consists of a sequence of instances with boxes, labels, etc. The flexibility of language cues enables the model to possess a certain degree of conversational ability, directly and effectively utilizing the language modality information of the large model and its inherent language and knowledge reasoning capabilities. The trained model can adjust the task objective and output format based on the user's response, and can flexibly accommodate more task formats such as visual relational reference.

[0037] (II) Data Sample

[0038] For tasks such as Human-Object Interaction Detection (HOI), Scene Graph Generation (SGG), and Task-Driven Object Detection (TDOD), existing technologies typically employ corresponding models. Different models focus on different features, such as human keypoints, object boundaries, and human-computer contact areas. Using the same feature source leads to poor performance, indicating conflicts between different tasks. Therefore, when dealing with visual relationship recognition, if a single model is used, different task heads, separate feature extractors, and multi-stage pipelines are usually employed.

[0039] To enable the use of a unified model for recognition and detection of the aforementioned various visual relationship tasks, this study specifically designed multiple schemes for constructing data samples to effectively train the model. First, this study treats visual relationship tasks as components of several basic and simple tasks, each including the following core elements: image (I), object location (O), object category (C), object function (A), presence or absence of interaction (B), and relationship category (R). These elements exhibit specific dependencies within a visual relationship task. By describing visual relationship tasks using task elements, the model does not directly handle multiple complex visual relationship tasks but rather processes some basic task elements, thereby reducing conflicts between different tasks and leveraging task similarity through these task elements.

[0040] Visual relationships are reconstructed based on task elements to obtain pseudo-data samples; real data samples and pseudo-data samples are used to train a multimodal large model, which can perform general visual relationship recognition and detection.

[0041] (2.1) Constructing data samples based on task element chains

[0042] like Figure 3 As shown, the left side represents the common elements and task element tree based on visual relationship task representation, while the right side represents the reorganization methods under different guidance.

[0043] The task is decomposed into a chain of task elements, and visual relationships are reconstructed using thought chains. That is, given a clear task cue indicating the prediction of the thought chain, the model predicts the elements in the task element chain before making the final prediction for that task. Specifically, based on the dependencies between task elements involved in different visual relationships, a sample that directly outputs the result of a complex visual relationship is transformed into a sample that sequentially outputs the corresponding results of different elements in a single-turn dialogue, and this sample is added to the training set. For example, the task chain decomposition task for HOI detection is I→O→C→B&A→R. Therefore, for HOI detection samples, the model should predict the object position (O), object category (C), presence or absence of interaction (B), and object function (A) sequentially before predicting the relationship category (R), thereby reducing the difficulty of reasoning about complex targets through the thought chain mechanism. See also Figure 3 The structure is split into chains.

[0044] The task is decomposed into a chain of task elements. These elements are output based on one or more rounds of dialogue prompts, with each round yielding a reconstructed visual relationship. Specifically, a sample that directly outputs a complex visual relationship is transformed into a sample that sequentially outputs the corresponding results of different elements in multiple rounds of dialogue, with each round corresponding to one element. This sample is then added to the training set. See also... Figure 3 Multiple rounds of splitting in the process.

[0045] The task is decomposed into a chain of task elements. A link between the input and output nodes of this chain is used as the reconstructed visual relationship. This transforms a sample that directly outputs a complex visual relationship into multiple samples corresponding to direct dependencies between different elements. Each sample represents a single-turn dialogue. These samples are then added to the training set. See also... Figure 3 The text is broken down into units.

[0046] This involves identifying the dependencies between task elements in real-world visual relationships, and then deriving indirect dependencies based on these dependencies. Specifically, it involves identifying all indirect dependencies that different task elements can form, constructing corresponding samples, and adding these samples to the training set. (See also...) Figure 3 The restructuring task in the process.

[0047] (2.2) Confidence-based pseudo-sample augmentation

[0048] To address the problem of poorly estimating the confidence (or uncertainty) of each predicted label in the output, this paper proposes guiding the trained model to evaluate confidence scores using language. By constructing partially correct pseudo-instances and associating them with confidence scores, the model can predict confidence scores after training, thus improving recall. Details are as follows.

[0049] First, for real visual relationships, if the acquired task elements include object position (0), object category (C), whether there is interaction (B), object function (A), and relationship category (R), set a confidence level for each object corresponding to object position (0), object category (C), and object function (A), and set an association confidence level for the two objects involved in whether there is interaction (B) and relationship category (R).

[0050] Next, by adjusting the detection boxes, the model can reposition objects, change object categories (C) or functions (A), pair objects related to interaction (B) or relationship categories (R), or perform random adjustments using object detection annotations in the dataset. The corresponding confidence scores are then estimated and updated to obtain new visual relationships. In these new visual relationships, the adjusted predicted target labels and the confidence scores of each target are merged into the predicted targets in the dialogue response, encouraging the model to predict more instances while simultaneously predicting confidence scores.

[0051] To improve the accuracy of the model's confidence estimation, an additional task is added to the model: Given any label provided in the question, predict the confidence value of that label in the answer.

[0052] The aforementioned positional adjustments involve fine-tuning the labels of the predicted targets in each dialogue response. This includes adjusting the bounding box coordinates, object category labels, and connectivity between objects. A confidence level is calculated based on the adjustment magnitude; the greater the deviation between the adjusted and unadjusted values, the lower the confidence level. Figure 4 For example, compared to real instances, the bounding box of the pseudo-instance containing an "umbrella" shifts significantly and has a low confidence score of 0.08. The purpose of this is to encourage the model to discover more instances with correct class labels and remain sensitive to inaccurate locations. This is suitable for tasks with object locations (0).

[0053] The category adjustment described above involves changing the label of an object's category (C) and object function (A). This is done by altering the category label of a real-world instance and calculating the semantic similarity between the new label and the real label. The confidence score is then readjusted based on this semantic similarity. The aim is to encourage the model to discover more instances with correct object location and relationship labels while remaining sensitive to incorrect categories. In practice, the adjusted predicted target labels and the confidence scores for each target are merged into the predicted target in the dialogue response, encouraging the model to predict more instances while simultaneously predicting confidence scores. This is suitable for tasks involving either object category (C) or object function (A).

[0054] The pairing adjustment described above concatenates two objects in an image that do not belong to a single real-world pair into a new pair in the training data. Since the relations (if any) in this pair are unannotated, the labels corresponding to the relation labels are simply masked during training. The purpose of this approach is to encourage the model to discover more pairs, especially since datasets are often not fully labeled, and some pairs may be missed. This is suitable for tasks involving interaction (B) or relation categories (R).

[0055] The above-mentioned random adjustment using object detection annotations in the dataset is a form of detection box supplementation, which can encourage the model to predict additional detection boxes, but the confidence score is low.

[0056] The pseudo-instances generated in the above manner are appended to the tails of real instances to form a target sequence for training. This method does not rely on "noisy" instances with random coordinates, but instead utilizes the compositional properties of visual relation tasks to generate four types of partially correct instances: correct labels have high confidence, and incorrect labels have low scores.

[0057] For each real instance, in one implementation, various types of pseudo-instances are generated with a certain probability. In another implementation, pseudo-instances are generated using a combination of two or three of the aforementioned position adjustment, category adjustment, and pairing adjustment methods. In yet another implementation, instances are generated randomly.

[0058] Furthermore, in some embodiments, if there are multiple true visual relationships in the task, an additional placeholder can be added after each true relationship category label to encourage the model to predict more relationships between each pair. Specifically, for tasks such as human-object interaction detection and scene graph generation, an additional placeholder is added after the true relationship category label. This placeholder does not calculate the loss during training; it only encourages the model to predict more relationship categories.

[0059] In some embodiments, if the task elements of the real visual relationships in the task include object features (A), and there may be multiple object features (A), then for tasks that include object features (A) as output, additional placeholders are added to encourage more predictions of object availability labels.

[0060] (2.3) Data Sample Organization

[0061] By designing different prompts and utilizing data organization methods such as open sets and weak supervision, the model can leverage larger-scale data and retain generalization ability during training. The steps include:

[0062] (2.3.1) Given possible category names in the dialogue question, the predicted category in the dialogue response is selected from the category names given in the question. The category used for inference is not necessarily the same as the one used in training. This is similar to an open vocabulary setup. It helps the model generalize to a finite and fixed vocabulary.

[0063] (2.3.2) For datasets that lack labels such as relation labels, such as object detection, the same task format as ordinary visual relation datasets is adopted. During training, the loss is not calculated for the part corresponding to the missing label, forming a similar weak supervision approach.

[0064] (2.3.3) The labels of the visual relationship detection data are reconstructed to form samples that include task forms such as object detection, image description, and region description, and these samples are added to the training set. It should be noted that the constructed training samples only introduce more task forms; no additional images or new annotations are added.

[0065] For example, the Described Object Detection (DOD) task is a form of object detection that uses descriptions. The description is input by concatenating labels corresponding to object category (C), presence or absence of interaction (B), object function (A), and relationship category (R), and the output label is the object location (0). For content without a description, no prediction is required.

[0066] An example is the image captioning task, applicable to any dataset. Simple caption sentences are constructed from HOI / SGG / visibility / detection annotations.

[0067] For example, the Grounded Captioning task is similar to image captioning, but only applies to a given region. It is applied to a dataset labeled with object locations (O). For each object location (O) as input, it takes labels for the relevant object category (C), whether there is interaction (B), object function (A), and relationship category (R) as the caption for prediction in the output.

[0068] From the above reconstruction of new visual relationships using labels corresponding to all visual relationships, it can be seen that this method establishes a correspondence between some task elements and other task elements.

[0069] (III) Experimental Comparison

[0070] To verify the practical performance of this application on various visual relationship recognition and detection tasks, it was tested on the well-known human-object interaction detection dataset V-COCO, the scene graph generation dataset Open Images V6, and the task-driven object detection dataset COCO-Tasks. Additionally, it was tested on the visual relationship referencing dataset VRD. Notably, this application was trained in a single run on the union of the training sets of these datasets and the training sets of the object detection datasets COCO and Object365, eliminating the need for dedicated fine-tuning on individual training sets.

[0071] The test sets for each dataset are as follows: The V-COCO dataset is a subset of COCO, featuring human-object interaction annotations. Its test set contains 4946 images, 80 object categories, and 29 interaction categories. For V-COCO, this study uses the common average precision as the metric. Open Images is a widely used scene graph generation dataset. Its test set contains 5322 images, 301 object categories, and 31 action categories. Following previous work, this study uses top-50 recall, relation-weighted average precision, phrase-weighted average precision, and a weighted composite score as metrics. COCO-Tasks is a task-driven object detection dataset with 14 tasks, each with 900 test images. It uses task average precision as the metric. VRD can be used as a dataset for visual relation referencing tasks; it uses average IoU as the metric.

[0072] Table 1 shows a comparison of the performance of this invention with existing methods on the test set of the V-COCO human-object interaction detection dataset. The present invention outperforms the best existing methods on this dataset.

[0073] Table 1

[0074]

[0075]

[0076] Table 2 presents a comparison of this invention with other methods that perform well on the Open Images scene graph generation dataset across various metrics. Similarly, in terms of the weighted score, a comprehensive metric, this invention surpasses the best existing technology on the Open Images dataset.

[0077] Table 2

[0078]

[0079] Similar to Tables 1 and 2, Table 3 presents a comparison of the average accuracy metrics between this invention and other state-of-the-art methods on the COCO Tasks dataset for task-driven object detection. This invention also achieves state-of-the-art results. The combined results from Tables 1, 2, and 3 demonstrate that the proposed method, using only a single model, simultaneously achieves state-of-the-art performance on various benchmarks across multiple different visual relationship tasks.

[0080] Table 3

[0081]

[0082] Table 4 presents a comparison between the proposed method and previous methods on the Visual Relationship Dependency (VRD) dataset. As can be seen from Table 4, for this new task format, the proposed method achieves performance close to that of previous fully supervised specialized models with zero samples, which well illustrates the generalization and flexibility of the model proposed in this invention.

[0083] Table 4

[0084]

[0085] As can be seen from Tables 1 to 4, this method achieved overall leadership across four tasks, four datasets, and a total of eight metrics, reaching the optimal level in the vast majority of metrics.

[0086] In summary, the proposed method has better performance on generalized visual relation tasks. It can not only handle various visual relation tasks that were previously handled by separate specialized models with a single model without fine-tuning, but also flexibly accommodate other task forms such as visual relation referencing, and achieve good zero-shot generalization on datasets such as VRD that were not seen during training.

[0087] In another embodiment, a general visual relationship recognition and detection system for a multimodal system is implemented according to the above method. The system includes a network architecture of a multimodal large model and a training sample construction scheme for visual relationship tasks.

[0088] If the above-described methods or systems are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the essence of the above-described technical solutions, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A general visual relationship recognition and detection method based on a multimodal large model, characterized by the following steps: include: Task elements are obtained based on real visual relationships in the task, and the task elements include relationship category, object position, object category, whether there is interaction, and object function; Visual relationships are reconstructed based on task elements to obtain pseudo-data samples; The multimodal large model is trained using real data samples and pseudo data samples. The trained multimodal large model can perform general visual relationship recognition and detection. The steps of reconstructing visual relationships based on task elements include: reconstructing visual relationships using thought chains based on task element chains; or outputting task elements according to one or more rounds of dialogue prompts based on task element chains, obtaining a reconstructed visual relationship in each round; or using a link between the input node and the output element node of the task element chain as a reconstructed visual relationship; or obtaining the dependency relationships of task elements in the real visual relationship, obtaining indirect dependency relationships based on the dependency relationships, and using the indirect dependency relationships to combine task elements to reconstruct visual relationships. The steps for obtaining the pseudo-data sample include: Data samples are constructed based on task element chains: the task is decomposed into task element chains, and visual relationships are reconstructed using thought chains; given a task prompt that explicitly indicates the prediction of the thought chain, the model predicts the elements on the task element chain before completing the final prediction of the task; according to the dependencies between task elements involved in different visual relationships, a sample that directly outputs the result of complex visual relationships is transformed into a sample that outputs the corresponding results of different elements in sequence in a single-turn dialogue, and this sample is added to the training set. Confidence-based pseudo-sample augmentation: First, for real visual relationships, if the acquired task elements include object location, object category, interaction, object function, and relationship category, a confidence score is set for each object corresponding to its location, category, and function. For objects involved in interaction or relationship categories, an association confidence score is set. Next, by adjusting the detection boxes, the object location is adjusted, or the object category or function is relabeled, or two objects involved in interaction or relationship categories are paired and adjusted, or random adjustments are made using object detection annotations in the dataset. The corresponding confidence scores are estimated and updated to obtain new visual relationships. In the new visual relationships, the adjusted predicted target labels and the confidence scores of each target are merged into the predicted targets in the dialogue response, encouraging the model to predict more instances while simultaneously predicting confidence scores.

2. The method according to claim 1, characterized in that: If there are multiple real visual relationships in the task, add an extra placeholder after each real relationship category label; If the task elements in the real visual relationships of the task include object functions, then add an extra placeholder after their corresponding labels; The placeholders are not included in the loss calculation during training.

3. The method according to claim 1, characterized in that, The steps of reconstructing visual relationships based on task elements include: providing a range of category names to be selected in the dialogue prompt.

4. The method according to claim 1, characterized in that, The steps of reconstructing visual relationships based on task elements include: reconstructing new visual relationships using the labels corresponding to all visual relationships, thereby establishing a correspondence between a portion of task elements and another portion of task elements.

5. The method according to claim 1, characterized in that: When performing visual relationship recognition and detection, the multimodal large model will... As input, For image, Given a text sequence, the detected visual relationships will be output as a text response. in: , For the number of conversations, and It is the first The text questions and text answers for this dialogue.

6. The method according to claim 5, characterized in that: The multimodal large model uses two image backbone networks of different sizes to extract image features. The output layer of the larger image backbone network is followed by a pooling layer to reduce the length of the corresponding features. The obtained image features are each processed by a multilayer perceptron to obtain two image feature sequences. The text feature sequence is obtained from the text sequence and merged with the image feature sequence into an image-text feature sequence, which is then input into the large language model. For each position in the image-text feature sequence, the large language model predicts the next lexical term based on all preceding lexical terms.

7. The method according to claim 6, characterized in that, The text feature sequence is obtained by processing the text sequence using a word segmenter.

8. A computer-readable storage medium, characterized in that: The computer program is stored that can be loaded by a processor and executed according to any one of claims 1 to 7.

Images