Construction method of scene graph question and answer inference model based on reinforcement learning

Abstract

The invention relates to the technical field of visual questions and answers, in particular to a scene graph question and answer inference model construction method based on reinforcement learning, which comprises the following steps: calling an existing large model interface, performing deep processing on an initial scene data set, and constructing a first batch of training sets; carrying out first-stage reinforcement learning training on a pre-constructed multi-modal large model by utilizing the first batch of training sets; obtaining an LLaVA-CoT data set, screening the LLaVA-CoT data set, and inputting the screened LLaVA-CoT data set into the multi-modal large model trained at the first stage to obtain a reasoning result; calling an existing large model interface to carry out accuracy evaluation and correction on the reasoning result of the multi-modal large model to obtain a second batch of training sets; and performing second-stage reinforcement learning training on the multi-modal large model by using the second batch of training sets to obtain a final scene graph question and answer inference model. According to the method, a staged and differentiated training strategy is adopted, the characteristic advantages of two batches of data are brought into full play, staged training is carried out on the large model, and the model performance is gradually improved.

Description

Technical Field

[0001] This invention relates to the field of visual question answering technology, and more specifically to a method for constructing a scene graph question answering reasoning model based on reinforcement learning. Background Technology

[0002] With the rapid development of machine learning and deep learning, multimodal learning has emerged, aiming to simulate the human mechanism for processing diverse information. By integrating heterogeneous data such as images, videos, audio, and text, it achieves more intelligent and human-like information understanding and interaction. Against this backdrop, large-scale multimodal models based on large language models have become a research and application hotspot, with large-scale visual understanding models attracting particular attention. In recent years, they have made breakthroughs in areas such as object recognition and scene parsing, with performance on some tasks approaching human levels, demonstrating strong technological potential and application prospects.

[0003] Early large-scale models and multimodal large-scale models had limitations in handling complex problems and struggled to deeply analyze the essence of issues. With technological advancements, models like OpenAI's o1 series pioneered the exploration of long-thinking-chain reasoning. These models engage in deep "thinking" before answering questions, breaking down complex problems and deducing step-by-step to generate long-thinking chains similar to search algorithms, significantly enhancing reasoning capabilities. Subsequently, many other models, including Baidu's Wenxin large-scale model, followed suit, acquiring "long-thinking-chain" capabilities and excelling at complex tasks such as logical reasoning. The development of long-thinking-chain reasoning is of great significance. It allows large-scale models to break through the previous "fast thinking" mode based on simple knowledge matching, simulating the human thought process when solving complex problems through multi-step, deep logical deduction, achieving an advancement from perception to cognition. This not only propels artificial intelligence towards a higher level of intelligence but also demonstrates enormous application potential in fields such as medical diagnosis, scientific research, and complex engineering planning, helping to solve various intractable problems in the real world. It has become a key development direction for large-scale models to approach human intelligence levels and achieve general artificial intelligence.

[0004] While long thought chain reasoning has developed to a relatively powerful level, many of its multimodal forms rely solely on single multimodal information and perform reasoning within the text, failing to fully demonstrate its thought process that integrates multimodal content. Therefore, how to leverage structured information such as scene graphs to perform long thought chain reasoning from the perspective of understanding and generating multimodal information, thereby significantly improving multimodal capabilities from multiple dimensions, has become a pressing technical problem for those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a method for constructing a scene graph question answering reasoning model based on reinforcement learning. Starting from the perspective of understanding and generating multimodal information, it utilizes the structured information of scene graphs to perform long thought chain reasoning, thereby significantly improving multimodal capabilities from multiple dimensions.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for constructing a scene graph question-answering reasoning model based on reinforcement learning includes the following steps:

[0008] Obtain the initial scene dataset, call the existing large model interface, and set prompt words to guide the existing large model to perform in-depth processing on the initial scene dataset to build the first batch of training sets;

[0009] The first phase of reinforcement learning training is performed on the pre-built multimodal large model using the first batch of training set;

[0010] Obtain the LLaVA-CoT dataset, filter it, and input it into the multimodal large model trained in the first stage to obtain the inference results;

[0011] The existing large model interface is called to evaluate the accuracy of the inference results of the multimodal large model, and the errors in the inference results are corrected in a targeted manner to obtain the reconstructed dataset. A two-stage inference process is then performed on the reconstructed dataset to realize intelligent derivation from the generalized scene graph to the targeted answer, and the second batch of training set is obtained.

[0012] The second batch of training set is used to conduct the second stage of reinforcement learning training on the multimodal large model. After the training is completed, the final scene graph question answering reasoning model is obtained.

[0013] Furthermore, the data structure for each data point in the first batch of datasets is as follows: [Image, Detailed Image Description, Brief Image Summary, List of Image Entities and Relationships, Image Scene Graph].

[0014] Furthermore, the process for filtering the LLaVA-CoT dataset includes:

[0015] Image type filtering: Select natural scene images suitable for scene graph representation, and remove charts, mathematical diagrams, structured images in the natural science field, and datasets whose main purpose is text recognition;

[0016] Question and answer type filtering: Filter out question and answer data with clear accuracy assessment methods, and prioritize retaining multiple-choice questions and open-ended simple-response questions;

[0017] After two rounds of selection, a subset of the LLaVA-CoT dataset was obtained. The data structure of each data point in the subset is as follows: [Image, User Question, Reasoning Process, Accurate Answer].

[0018] Furthermore, the process of the multimodal large model trained in the first stage performing inference on the selected LLaVA-CoT dataset includes:

[0019] Adjust the generation parameters of the multimodal large model, and randomly output four inference results for each data point in the subset. The data structure of each data point in the subset is expanded to: [image, four image inference and scene graph results, user question, inference process, accurate answer].

[0020] Furthermore, the process of accuracy evaluation and correction of the inference results of multimodal large models includes:

[0021] Call the existing large model interface, set prompt words, and evaluate the accuracy of each part of the inference result of the multimodal large model for each data point;

[0022] If all parts of the reasoning result of a certain data are determined to be accurate, then it is directly included in the final reconstructed dataset;

[0023] If all four inference results corresponding to a certain data point contain errors, the number of correct parts in each inference result will be automatically calculated, and the inference result with the highest accuracy will be selected to enter the reconstruction process. During reconstruction, the existing large model interface will be called, and prompts will be set to guide the existing large model to make targeted corrections to the erroneous parts. After correction, the model will be included in the reconstructed dataset.

[0024] The data structure for each data point in the reconstructed dataset is as follows: [Image, detailed description of the image, brief summary of the image, list of image entities and relationships, image scene diagram, user question, reasoning process, accurate answer].

[0025] Furthermore, the two-stage inference process performed on the reconstructed dataset includes a scene subgraph extraction inference process and a multimodal inference generation process; wherein, the scene subgraph extraction inference process includes:

[0026] Call the existing large model, set prompt words, and semantically align user questions with entities and relationships in the scene graph;

[0027] Extracting relevant entities and triples based on question keywords;

[0028] Construct targeted scenario subgraphs to form a structured representation that is strongly related to the problem. The final output format is JSON, which contains the entities and relationships that the problem focuses on.

[0029] The multimodal reasoning generation process includes:

[0030] By utilizing the extracted scene subgraphs and original information, the existing large model performs multi-round reasoning generation: integrating multi-source information such as scene subgraphs, detailed descriptions, and user questions;

[0031] A thought chain reasoning strategy is employed to generate intermediate reasoning steps;

[0032] Apply constraint satisfaction mechanisms to ensure logical consistency;

[0033] The final output includes an answer text and explanation process that conforms to academic standards.

[0034] Furthermore, after calling the existing large model interface to perform a two-stage inference process on the reconstructed dataset, it also includes:

[0035] A dual verification mechanism is used to verify the generated final answer. For multiple-choice questions, the accuracy is judged directly by matching the options.

[0036] For open question-and-answer type data, in addition to basic rule judgment, the semantic understanding capabilities of the existing large model are also introduced for secondary verification;

[0037] If the final verification result shows that the answer is inaccurate, the retry mechanism will be automatically triggered to re-call the existing large model interface to regenerate the data until the accurate answer is obtained or the preset retry limit is reached.

[0038] The final data structure for each item in the second batch of training sets is as follows: [Image, detailed image description, brief image summary, list of image entities and relationships, image scene graph, scene graph extraction and reasoning, scene subgraph, reasoning, user question, accurate answer].

[0039] Furthermore, in the first phase of reinforcement learning training, a format reward function and an accuracy reward function are constructed. The format reward function constrains the output of the multimodal large model from three dimensions: generation step format reward, length constraint reward, and scene graph format reward, so that the generated scene graph conforms to the predetermined format standard.

[0040] The accuracy reward function constrains the output of the multimodal large model from two dimensions: scene graph relationship reward and scene graph entity reward, focusing on the accuracy of entity and relationship recognition in the scene graph.

[0041] Furthermore, in the second phase of reinforcement learning training, a multi-type data hybrid training strategy is adopted, covering scene graph generation inference data, scene graph open question answering inference data, and scene graph multiple-choice question answering inference data.

[0042] For training the inference data generated from the scene graph, the evaluation mechanism of the scene graph accuracy reward function from the first stage is used.

[0043] For training on open question-and-answer reasoning data or multiple-choice question-and-answer reasoning data in scene graphs, an absolute content matching combined with rule-based judgment is used. By accurately comparing the answer content with the established rules, the validity of the multimodal large model output is determined.

[0044] Furthermore, the existing large model interface invoked is either the GPT-4o large model or the Gemini large model.

[0045] As can be seen from the above technical solution, compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. This invention integrates multimodal understanding and generation into the reasoning process through scene graphs, which can transform multi-dimensional information such as images and vision into a structured "entity-relationship" network. When multimodal large models are dealing with complex tasks, they can reason step by step along the logical chain of the scene graph. This not only achieves deep fusion of multimodal information, but also gradually generates the structured network information that the model is currently interested in. This information can be used to further refine reasoning and accurately locate the problem, thereby improving the credibility of the model's reasoning process, the robustness of the model's generated reasoning content, and the accuracy of the final result.

[0047] 2. This invention employs a step-by-step training model to construct scene graph reasoning question-answering data. Addressing the current lack of datasets that simultaneously contain scene graphs and reasoning question-answering annotations, the step-by-step training strategy becomes a key breakthrough. This invention innovatively breaks down the data construction process into two stages: The first stage trains a multimodal large model to generate scene graphs, transforming raw visual multimodal data into structured scene graphs, enabling explicit extraction of entity relationships. The second stage, combined with existing reasoning question-answering datasets, uses techniques such as fusion, rewriting, and enhancement on the scene graphs generated by the large model trained in the first stage to construct novel training data that contains both scene graph structural information and question-answering reasoning logic. This step-by-step construction method not only fills the gap in specific data formats but also allows the large model to establish a deep connection between the logical context of the scene graph and the semantic requirements of the question-answering task, providing high-quality and suitable training samples for subsequent long-chain reasoning based on scene graphs.

[0048] 3. This invention utilizes a hybrid training mode of fine-tuning and reinforcement learning, employing a training model that combines R1 supervised fine-tuning (SFT) with general relativistic policy optimization (GRPO), demonstrating unique advantages in training multimodal reasoning models. R1 SFT optimizes the pre-trained model based on a small amount of high-quality labeled data, quickly calibrating the model output to fit specific task requirements and avoiding the resource waste and inefficiency of training from scratch. Meanwhile, the GRPO method dynamically evaluates the model's decision paths in complex reasoning tasks, adjusting the reward mechanism with general relativistic optimization principles, making the model more focused on key aspects of long-chain reasoning, enhancing the coherence and accuracy of reasoning logic. The combination of these two methods achieves efficient adaptation during the "cold start" phase through SFT and dynamically optimizes the model strategy during reinforcement learning through GRPO, forming a closed-loop training system of "precise targeting - dynamic reinforcement." This significantly reduces the data scale and computational resources required for training and significantly improves the model's generalization ability and efficiency in handling complex tasks in multimodal scene graph reasoning. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0050] Figure 1 A flowchart illustrating the method for constructing a scene graph question-answering reasoning model based on reinforcement learning provided by this invention;

[0051] Figure 2 A flowchart illustrating the construction process of the second batch of training sets provided by this invention;

[0052] Figure 3 The design flowchart of the format reward function in the first stage of reinforcement learning provided by this invention;

[0053] Figure 4 The flowchart for the design of the accuracy reward function in the first stage of reinforcement learning provided by this invention;

[0054] Figure 5 The design flowchart of the format reward function in the second stage of reinforcement learning provided by this invention;

[0055] Figure 6 The recognition result is shown in Example 1;

[0056] Figure 7 The scene diagram and scene sub-diagram for Example 1;

[0057] Figure 8The recognition result is shown in Example 2;

[0058] Figure 9 This is the scene graph and scene subgraph for Example 2. Detailed Implementation

[0059] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0060] Before introducing the technical solution of this invention, the meanings of some technical terms will be explained.

[0061] Scene Graph: A scene graph is a data structure used to encode the semantic information of an image. It uses a graph model to depict objects and their relationships within an image, accurately capturing the complex semantics and structure of the image. In a scene graph, nodes correspond to entities in the image, including objects, people, or identifiable scene elements. Each node carries a category label to clearly define the entity type. Edges connect nodes, representing semantic connections such as physical associations (e.g., "next to") and logical relationships (e.g., "belong to"). Relationship labels serve as descriptors for the edges, intuitively defining the specific relationships between nodes, such as "a child is standing under a tree." These three elements work together to construct a structured representation of the image content, providing a clear semantic framework for computer vision tasks.

[0062] CoT (CoMatching Logic) is a prompting technique for large language models designed to optimize their performance on complex reasoning tasks. By introducing intermediate reasoning steps between questions and answers, CoT simulates the step-by-step, progressive thinking process of humans, allowing models to break down complex tasks into multiple sub-tasks for solution. This significantly improves accuracy in arithmetic, common sense, and symbolic reasoning tasks. Furthermore, CoT makes the model's reasoning process explicit, transforming its output from a "black box" into a more interpretable and reliable representation. This helps users better understand the model's decision-making rationale and provides crucial insights for subsequent optimization and iteration.

[0063] Command fine-tuning: Command fine-tuning of large models refers to the process of enabling pre-trained models to understand and execute various user commands through supervised learning. Specifically, it involves first collecting a dataset of commands covering multiple domains and tasks (such as question answering, translation, and text generation), with each dataset containing a command description and corresponding expected output. This data is then used to specifically train the pre-trained model, adjusting its parameters to generate expected responses based on input commands. This process transforms the model from a general knowledge store into a task-oriented agent, significantly improving its practicality in real-world scenarios. Common methods include full-parameter fine-tuning and efficient parameter fine-tuning (such as LoRA and PEFT). The former directly updates all model parameters, while the latter reduces costs by freezing most parameters and adjusting only a small number of adapter parameters. Command fine-tuning is a crucial step in transforming large models from "general intelligence" to "specialized capabilities," and it is also a core technical step in current industrial applications.

[0064] Reinforcement Learning: Reinforcement learning is a machine learning paradigm where agents learn to make decisions through interaction with their environment. The agent selects actions based on the environmental state, receives rewards and new states in return, and gradually finds the optimal policy by maximizing accumulated rewards. Proximal Policy Optimization (PPO), proposed by OpenAI in 2017, is a cutting-edge algorithm in reinforcement learning. It avoids the performance degradation caused by excessive policy updates in Trust Region Policy Optimization (TRPO) in a simpler and more efficient way. PPO modifies the loss function and uses a clipping mechanism to constrain the difference between the old and new policies, ensuring that the new policy iteratively optimizes near the old policy. It is efficient, stable, easy to implement, and highly applicable, and is widely used in various reinforcement learning tasks, especially in Human Feedback-Based Reinforcement Learning (RLHF), where it is often used to fine-tune pre-trained language models to improve the fit between content generation and human preferences. Group Relative Policy Optimization (GRPO), an online learning algorithm proposed by DeepSeek, was initially used to fine-tune R1 and R1-Zero models, focusing on enhancing the inference capabilities of large language models (LLMs). GRPO breaks away from the traditional model that relies on external critic evaluation, optimizing the model through mutual evaluation of group responses. When calculating advantage, it samples actions and normalizes rewards, eliminating the need for a value network and reducing computational and storage costs. In the policy update phase, it precisely controls the magnitude of policy updates by introducing a KL divergence term into the loss function. Compared to PPO, GRPO is more computationally efficient and better suited for fine-tuning large-scale language models.

[0065] o1 / r1 model: OpenAI's o1-type model focuses on enhancing reasoning ability. It uses large-scale reinforcement learning to optimize thinking processes, correct errors, and decompose complex tasks. It has performed outstandingly in the fields of science, coding, and mathematics. For example, its success rate in the International Mathematical Olympiad qualification exam exceeds that of GPT-4o. It has also spawned o1-preview and the lower-cost o1-mini version, which are suitable for scientific research, software development, and other scenarios.

[0066] DeepSeek's R1-class model uses reinforcement learning to achieve reasoning chains of tens of thousands of words. Its performance rivals that of O1 in math competitions and programming assessments. It has been locally deployed in government and enterprise scenarios such as Hohhot and Honor, and applied to areas like official document writing and intelligent question answering. Both models focus on optimizing the model's thought process, autonomously correcting errors, and using reasoning to solve complex problems.

[0067] Accuracy, recall, and F1 score: In the fields of machine learning and information retrieval, accuracy, recall, and F1 score are key metrics for evaluating model performance. Accuracy measures how precisely a model predicts positive examples, i.e., the proportion of actual positive examples among samples predicted as positive. Recall, also known as the full detection rate, reflects the model's ability to capture actual positive examples, i.e., the proportion of actual positive examples correctly predicted. The F1 score is the harmonic mean of accuracy and recall, achieving a balance between the two. It is particularly suitable for scenarios with imbalanced samples, providing a more objective evaluation of the model's overall performance by combining the performance of both metrics.

[0068] The core technical idea of ​​this invention is that the data construction and cleaning process and the model training process are not independent of each other, but deeply intertwined and synergistic. The data construction and cleaning process provides a high-quality, structured data foundation for model training, while the model training process, through a feedback optimization mechanism, feeds back into the data construction, making it more aligned with the model's learning needs, thereby achieving a two-way improvement in data quality and model performance.

[0069] Current mainstream scene graph datasets and visual localization datasets, such as PSG and VG, primarily focus on entity location annotation and relation extraction, lacking question-answering information specific to image content. While datasets like LLaVA-CoT have made breakthroughs in image content reasoning, they are not associated with corresponding scene graphs. Neither of these datasets can simultaneously meet the dual requirements of structured representation of image scene graphs and question-answering reasoning.

[0070] Based on this, this invention proposes a solution based on large model construction and iterative training, successfully creating a novel dataset. This dataset not only achieves accurate mapping between scene graphs and images, but also incorporates question-answering reasoning elements, providing higher-quality data support for visual reasoning research.

[0071] like Figure 1As shown in the figure, this invention discloses a method for constructing a scene graph question-answering reasoning model based on reinforcement learning, including the following steps:

[0072] S1. Obtain the initial scene dataset, call the existing large model interface, and set prompt words to guide the existing large model to perform in-depth processing on the initial scene dataset to build the first batch of training sets;

[0073] S2. Use the first batch of training sets to perform the first stage of reinforcement learning training on the pre-built multimodal large model;

[0074] S3. Obtain the LLaVA-CoT dataset, filter it, and input it into the multimodal large model trained in the first stage to obtain the inference results;

[0075] The existing large model interface is called to evaluate the accuracy of the inference results of the multimodal large model, and the errors in the inference results are corrected in a targeted manner to obtain the reconstructed dataset. A two-stage inference process is then performed on the reconstructed dataset to realize intelligent derivation from the generalized scene graph to the targeted answer, and the second batch of training set is obtained.

[0076] S4. Use the second batch of training set to perform the second stage of reinforcement learning training on the multimodal large model. After the training is completed, the final scene graph question answering reasoning model is obtained.

[0077] Next, we will elaborate on the key steps, core strategies, and optimization logic of data construction and cleaning in detail.

[0078] S1. Construct a high-level reasoning scenario graph to generate a dataset, i.e., the first batch of datasets.

[0079] Based on a high-quality initial scene graph dataset, an image-scene graph inference link is constructed in reverse. The original data is presented in the form of image-scene graph pairs and stored in JSON format.

[0080] The content of the scene diagram adopts <sg>< / sg> The image objects are represented in a structured manner using JSON format wrapped in tags. The Objects field stores image objects as a dictionary, with each entity key-value pair corresponding to the entity name and the coordinates of the detection box, respectively. The Triplets field records the relationships between objects in the form of triples, with the middle element explicitly specifying the entity association attributes.

[0081] To obtain highly refined scene graph information, a progressive reasoning process is adopted: First, the model generates detailed text descriptions based on the user-input image, effectively expanding the contextual semantic information; second, through semantic analysis, key content is extracted from the detailed descriptions to accurately locate the core entities and their relationships in the image; subsequently, the model categorizes and organizes the extracted entities and relationships, constructing entity lists and relationship lists respectively; finally, the above information is integrated into a standardized structured scene graph. Specifically, the first step of data construction generates data in the following format:

[0082] Detailed caption for the image;

[0083] <summary> Summary based on the caption< / summary> ;

[0084] <content> Each object and the relationship between them< / content> ;

[0085] <sg> The scene diagram generated based on the above caption and reasoning.< / sg> .

[0086] in <content>Some of these can be extracted from the scene graph content of the original data and presented in the following format:

[0087] Objects are:

[0088] 1.name:curtain-01; bounding box:[0.0,0.0,0.18,1.0];

[0089] 2.name:person-01; bounding box:[0.0,0.18,0.98,0.99];

[0090] 3.name:tie-01; bounding box:[0.45,0.71,0.55,1.0];

[0091] 4.name:wall-01; bounding box:[0.12,0.0,1.0,1.0];

[0092] Relations are:

[0093] 1.person-01 attached to tie-01;

[0094] 2.curtain-01 attached to wall-01;

[0095] 3. person-01 in front of wall-01;

[0096] 4.person-01wearing tie-01.

[0097] The other two key parts and <summary>The construction relies on a high-performance multimodal large model.

[0098] When generating detailed descriptions of images, GPT-4o is used as the core tool. Precisely designed prompts guide the model to perform a comprehensive and multi-layered analysis of the image content. This process ensures that the generated descriptions cover multiple dimensions of the image, including key elements, spatial relationships, and color features, providing rich and detailed textual information for subsequent research and applications.

[0099] for <summary>The construction of this part relies on the powerful semantic understanding and generalization capabilities of a large model. Based on the structured information of the scene graph and the visual content of the image, it generates summary text that combines key information extraction with concise expression. This invention aims to accurately extract the core elements of the image while providing a concise and succinct description of the image content. In the actual data construction process, the model is guided to complete this task by calling the GPT-4o interface and using carefully designed prompts.

[0100] Ultimately, guided by previously designed prompts, this invention utilizes GPT-4o to perform deep processing on an existing scene graph dataset, successfully constructing 5000 high-quality data entries. This data is specifically designed for fine-tuning the pre-built multimodal large model of this invention, aiming to endow the multimodal large model with the core capabilities of accurate reasoning and scene graph generation. The final constructed data is presented in the form of multi-source information fusion, containing five key elements. The specific data structure of each data entry is as follows: [Image, detailed image description, brief image summary, list of image entities and relationships, image scene graph].

[0101] S2. Use the first batch of training sets to perform the first stage of reinforcement learning training on the pre-built multimodal large model.

[0102] This invention trains the model using the first batch of training sets constructed in step S1, and employs instruction fine-tuning techniques for targeted training. Through designed instruction templates and task examples, the model is guided to learn and master the ability to generate responses according to a specified format, enabling it to accurately output structured and standardized results. This stage establishes a basic response framework and format understanding capability for the model. After multiple rounds of training and optimization, a multimodal large-scale model with scene graph reasoning generation capabilities has been successfully developed. This model demonstrates superior performance in integrating visual semantic understanding and structured knowledge representation.

[0103] Specifically, this invention utilizes non-inference data for reinforcement learning training. To achieve the dual goals of efficient training and stable optimization, the GRPO (Generalized Rejection Policy Optimization) method is introduced, which features efficient optimization with few samples, excellent training performance, and strong parameter stability. During training, the model's reward function is scientifically designed to precisely guide the model in learning inference logic and decision-making strategies, thereby further improving the model's inference ability and generalization performance in complex tasks.

[0104] In the first stage of reinforcement learning training, a two-dimensional reward function system of format constraint and accuracy verification was constructed. By scientifically weighting the format reward function and the accuracy reward function, and then weighting and summing them, the final reward function was used to guide the model to generate high-quality output. The specific content is as follows:

[0105] 1) Formatted reward function: Primarily used to standardize the data generation structure; the design process is as follows: Figure 3 The function constrains the model output from three dimensions: generation step format reward, length limit reward, and scene graph format reward, ensuring that the generated data conforms to the established format standards.

[0106] ① Generation Step Format Rewards: For each data entry, detail descriptions of each part are required.

[0107] <summary>(Summarize), <content>(Inference content) <sg>Core modules such as (scene diagram) use positive incentives to encourage the model to output in a specified format, thus avoiding structural chaos.

[0108] ② Length limit reward: Dynamically balance the level of detail in the data to prevent the model from generating redundant information (such as excessively large scene diagrams or too many irrelevant entities) while avoiding overly simplified descriptions that lead to the loss of key information.

[0109] ③ Scene graph format reward: Force the model to generate scene graphs in a standardized JSON structure, requiring that they uniformly include Objects and Triplets fields, and make precise constraints on the data format: the bounding box coordinates of each entity in Objects must include 4 dimensions (x1, y1, x2, y2); the length of the relationship list in Triplets is fixed at 2, to ensure the standardization of structured expression.

[0110] Weighting: Considering that the model has already mastered the basic format through early supervised fine-tuning, the format reward weight for the generation step is set to 0.3; the length limit reward is equally important, and its weight is also 0.3; while the structured accuracy of the scene graph has a significant impact on downstream tasks, so it is given the highest weight of 0.4.

[0111] 2) Accuracy Reward Function: This function constrains the output of the multimodal large model from two dimensions: scene graph relationship reward and scene graph entity reward. It focuses on the accuracy of entity and relationship recognition in the scene graph, ensuring the quality of scene graph generation. The design process is as follows: Figure 4 As shown.

[0112] ① Scene graph relationship reward: A hierarchical matching strategy is used to calculate the relationship accuracy. The semantic similarity of action words in the relationship triples is compared first. If the action matching degree meets the standard, the semantic similarity of entity words is further calculated. For the successfully matched triples, the overlap of entity detection boxes is judged by the IOU threshold (>0.6). Finally, the F1 score is used as the comprehensive evaluation index.

[0113] ② Scene graph entity reward, with independent evaluation centered on entities: Extract the entity list from the generated scene graph and the standard answer, and compare the semantic similarity of entity names with the overlap rate of detection boxes (IOU>0.6) in turn. The F1 score is also used to measure the overall recognition accuracy.

[0114] Weighting: Given that entity recognition is the foundation of scene graph construction, it is given a high weight of 0.7; while the weight of relationship recognition is set to 0.3 to highlight the difference in importance between the two.

[0115] Through reinforcement learning training using the aforementioned reward function system, the model achieved significant improvements in both format regularity and content accuracy (detailed experimental data can be found in the Results Data section). The optimized model was then applied to subsequent data construction work to further ensure data quality and inference performance.

[0116] S3, Construction of the second batch of training sets.

[0117] In this step, the LLaVA-CoT dataset is first obtained, and then subjected to two rounds of screening. The specific screening process includes:

[0118] Image type selection: Select natural scene images suitable for scene graph representation, and remove charts, mathematical diagrams, structured images in the natural science field, and datasets with text recognition as the main target, to ensure that the image data is suitable for scene graph construction.

[0119] Question and answer type filtering: Filter out question and answer data with clear accuracy assessment methods, and prioritize retaining multiple-choice questions and open-ended simple response questions to lay a solid data foundation for subsequent quantitative accuracy assessment and reinforcement learning training.

[0120] After two rounds of selection, a subset of the LLaVA-CoT dataset was obtained. The data structure of each data point in the subset is as follows: [Image, User Question, Reasoning Process, Accurate Answer].

[0121] After the two rounds of screening, a carefully selected subset of the LLaVA-CoT dataset was formed. This batch of data focuses on scene graph deep reasoning scenarios, encompassing a series of question-answer pairs that require logical deduction to answer, and ensuring that each sample contains corresponding scene graph information, forming an organic combination of structured knowledge and reasoning tasks. Each data structure contains four core elements: image, user question, reasoning process, and accurate answer.

[0122] Based on the selected LLaVA-CoT dataset, the multimodal large model trained in the first stage is used to generate scene graphs from the image data in this dataset. By setting the generation parameters do_sample=True, top_p=0.8, and temperature=1.0, the randomness and creativity of the large model are stimulated, enabling diversified generation of reasoning processes and scene graphs. For each data point in the subset, the model will randomly output four reasoning results. The new data structure constructed in this way expands to [image, four image reasoning and scene graph results, user question, reasoning process, and accurate answer].

[0123] To further improve data quality, deep screening and cleaning were performed by calling existing large model interfaces (such as the GPT-4o or Gemini large model). A data-by-data judgment mechanism was adopted, where the large model evaluated the accuracy and corrected errors in each part of the inference results of each response. The specific accuracy evaluation and correction process included:

[0124] The existing large model interface is invoked, prompt words are set, and for each data point, the accuracy of each part of the inference result of the multimodal large model is evaluated. After obtaining the accuracy evaluation of each response from the large model, a strict stratified screening mechanism is executed:

[0125] ① Fully accurate response retained: If all parts of the reasoning result for a certain data (including detailed description, etc.) are retained... <summary>Summary <content>Content reasoning and <sg>If all scene diagrams are deemed accurate, they are directly included in the final reconstructed dataset.

[0126] ② Optimal response selection: If all four inference results corresponding to a certain data point contain errors, the number of correct parts in each inference result will be automatically calculated, and the inference result with the highest accuracy will be selected to enter the reconstruction process. During reconstruction, the existing large model interface will be called, and prompt words will be set to guide the existing large model to make targeted corrections to the erroneous parts. After correction, it will be included in the reconstructed dataset.

[0127] After data filtering and correction, a high-quality reconstructed dataset was obtained. The data structure of each data point is as follows: [Image, detailed description of the image, brief summary of the image, list of entities and relationships in the image, image scene diagram, user question, reasoning process, accurate answer].

[0128] Subsequently, based on the reconstructed dataset, a two-stage reasoning process was designed to achieve intelligent derivation from a generalized scenario graph to a targeted answer, specifically including:

[0129] ① Scene Subgraph Extraction and Reasoning: This stage calls the existing large model interface, sets prompt words, and semantically aligns the user's question with entities and relationships in the scene graph; extracts relevant entities and triples based on question keywords; constructs a targeted scene subgraph to form a structured representation strongly related to the question, and the final output format is as follows. <subsg>The JSON structure wrapped in tags contains the entities and relationships that the issue focuses on.

[0130] ② Multimodal reasoning generation: Utilizing the extracted scene subgraphs and original information, the existing large model performs multi-round reasoning generation: integrating multi-source information such as scene subgraphs, detailed descriptions, and user questions; adopting a chain-of-thought reasoning strategy to generate intermediate reasoning steps; applying a constraint satisfaction mechanism to ensure logical consistency; and finally outputting answer text and explanation process that conform to academic standards.

[0131] The final data format is:

[0132] Detailed caption for the image;

[0133] <summary>Summary based on the caption;

[0134] <content>Each object and the relationships between them;

[0135] <sg>The scene diagram <\SG> generated based on the above caption and reasoning;

[0136] <extraction>Based on the user's question and the scene diagram, locate the scene sub-diagram <\EXTRACTION>;

[0137] <subsg>The generated scene subgraph <\SUBSG>;

[0138] <think>Freely apply your reasoning based on user questions and answers, image content, and scene graph content, continuously updating the relationship information between nodes in the scene graph.

[0139] <answer>Final answer.

[0140] After calling the existing large model interface to perform a two-stage inference process on the reconstructed dataset, a rigorous dual verification mechanism was also established to verify the generated final answer:

[0141] For multiple-choice questions, accuracy is determined directly by matching the answer choices.

[0142] For open question-and-answer type data, in addition to basic rule judgment, the semantic understanding ability of the existing large model is also introduced for secondary verification; for example, if the standard answer is "woman" and the answer generated by the large model is "person1", even if the rule judgment may not match, the answer is still considered valid if it is confirmed as a reasonable answer by the semantic analysis of the large model.

[0143] If the final verification result shows that the answer is inaccurate, the retry mechanism will be automatically triggered to re-call the existing large model interface to regenerate the data until an accurate answer is obtained or the preset retry limit (e.g., 8 times) is reached.

[0144] The final second batch of training data contains rich multimodal information and reasoning processes. The data structure is as follows: [Image, detailed image description, brief image summary, list of image entities and relationships, image scene graph, scene graph extraction reasoning, scene subgraph, thinking reasoning, user question, accurate answer].

[0145] The data constructed above will be deeply integrated with the results of the first phase, jointly providing support for the training of the large model in the second phase. The overall data construction process can be summarized into two core steps:

[0146] First, a scene graph data augmentation strategy is implemented based on the large model interface. Through this strategy, the model can systematically extract and process information from image content according to predetermined steps, gradually transforming visual information into a structured scene graph representation, laying the foundation for subsequent inference.

[0147] Second, the large-scale model trained in the first stage is used to generate scene graph reasoning data containing question-and-answer text. Building upon this, the data is further enhanced using the existing large-scale model interface, guiding the model to accurately focus on key content based on scene graph information and to conduct deep reasoning in conjunction with the context, ultimately generating logically rigorous, accurate, and detailed answers. These two stages complement each other, jointly constructing a complete data construction system, providing high-quality, multi-layered data support for the training of the visual scene graph reasoning large-scale model.

[0148] By adopting a phased and differentiated strategy, we can give full play to the characteristics and advantages of the two batches of data, train the large model in stages, and gradually improve the model performance.

[0149] S4, the second phase of reinforcement learning training.

[0150] After completing the second phase of data construction, the instruction fine-tuning and reinforcement learning training process will be repeated to further optimize model performance. The reinforcement learning phase employs a multi-type data hybrid training strategy, specifically covering: scene graph generation and inference data, scene graph open-ended question-answering inference data, and scene graph multiple-choice question-answering inference data. To address the training objective of this phase, a reward function other than the scene graph reward function was designed to measure the accuracy of the model's generated answers.

[0151] The modified format reward function maintains a high degree of consistency with the format reward function used in the first stage of reinforcement learning training. It sets the format reward weight for the generation step to 0.3, the length constraint reward weight to 0.3, and the scene graph format reward weight to 0.4, continuing the strict constraints on the model output standardization. The process is as follows: Figure 5 As shown.

[0152] In lack Figure 4 When generating special rewards from the scene diagram shown, the evaluation of the generation quality mainly depends on the accuracy of the model generation.

[0153] For training the scene graph generation inference data, the evaluation mechanism of the scene graph accuracy reward function and the modified format reward function from the first stage is used to perform quantitative scoring from dimensions such as entity recognition and relationship matching.

[0154] For training on open question-and-answer reasoning data or multiple-choice question-and-answer reasoning data in scene graphs, an absolute content matching combined with rule-based judgment is used. By accurately comparing the answer content with the established rules, the validity of the multimodal large model output is determined.

[0155] After reinforcement learning training, the model achieved significant breakthroughs in both accuracy and scene graph generation capabilities. An innovative strategy combining multi-step data construction with two-stage training successfully activated the model's cold-start capability in scene graph reasoning, not only efficiently producing high-quality data conforming to the target format but also endowing the model with powerful reasoning performance, significantly improving the credibility of the output results.

[0156] At a deeper level, the model completes reasoning tasks by generating entirely new information related to images, a process that fully demonstrates its deep understanding and flexible application of multimodal information. The model organically integrates and internalizes multimodal information such as images and text as the basis for reasoning, making the reasoning chain more controllable and precise. This achievement not only strengthens the model's reasoning ability but also represents a significant breakthrough and improvement over cutting-edge models like O1 / R1 in terms of modal expansion.

[0157] To visually demonstrate the model's practical application effectiveness, two typical inference cases are provided. These cases are based on the final trained model and achieve accurate responses to user questions through scene graph analysis.

[0158] Example 1 emphasizes the model's scene graph recognition capabilities in complex scenarios. Leveraging stable and robust scene graph parsing technology, the model can quickly and accurately capture key information in images. Even when faced with interference such as blurriness and occlusion, it can still complete high-quality inference tasks, demonstrating excellent scene adaptability. The recognition results are as follows: Figure 6 As shown, Figure 6 The left image in the diagram is the user-provided image, and the right image is the bounding box corresponding to the entity in the recognized scene image. The generated scene image and scene sub-images are as follows: Figure 7 As shown, the red nodes represent the scene subgraph.

[0159] Example 2 focuses on the model's deep understanding and application of entity relationships: through detailed analysis of entities and their relationships in an image, the model can accurately match this information with user questions, and then generate accurate and reasonable answers through logical reasoning, highlighting its powerful performance in semantic understanding and knowledge association. The recognition results are as follows: Figure 8 As shown, Figure 8 The left image in the image is the user-provided image, and the right image is the bounding box corresponding to the entity in the scene image. The generated scene image and scene sub-image are shown below. Figure 9 As shown, the red nodes represent the scene subgraph.

[0160] Next, the performance of the finally trained scene graph question answering reasoning model of this invention will be evaluated, and the comparison results are shown in Table 1 and Table 2.

[0161] Table 1 Evaluation of the model's ability to extract entities from the generated scene graph.

[0162] DET Accuracy Recall rate F1 score VILA-SFT-25K 61.35％ 63.58％ 62.45％ Qwen2-VL-SFT-10k 73.97％ 70.46％ 72.17％ Qwen2-VL-grpo-10k-ng16 74.32％ 70.51％ 72.36％ Qwen2-5-VL-SFT-4.1k-ep4 72.26％ 66.96％ 69.51％ Qwen2-5-VL-SFT--grpo-1000s 77.28％ 65.12％ 70.68％ Qwen2-5-VL-SFT-25k-ep4 76.28％ 70.48％ 73.26％ Qwen2-5-VL-SFT-sg-qa-ep2 76.87％ 68.09％ 72.21％ Qwen2-5-VL-SFT-sg-qa-ep4 75.99％ 69.92％ 72.83％

[0163] Table 2 Evaluation of the Model's Ability to Extract Relationships from the Generated Scene Graph

[0164] SGDET Accuracy Recall rate F1 score VILA-SFT-25K 33.80％ 33.41％ 33.60％ Qwen2-VL-SFT-10k 40.07％ 35.18％ 37.46％ Qwen2-VL-grpo-10k-ng16 40.50％ 35.54％ 37.86％ Qwen2-5-VL-SFT-4.1k-ep4 39.88％ 34.20％ 36.82％ Qwen2-5-VL-SFT--grpo-1000s 43.44％ 33.76％ 37.99％ Qwen2-5-VL-SFT-25k-ep4 42.19％ 35.51％ 38.56％ Qwen2-5-VL-SFT-sg-qa-ep2 41.66％ 35.35％ 38.25％ Qwen2-5-VL-SFT-sg-qa-ep4 41.53％ 35.27％ 38.14％

[0165] DET refers to entity detection in the scene graph. It uses IOU to determine the overlap between predicted entities with similar entity names and the actual answer, and considers IOU greater than 0.6 as accurate recognition. SGDET, on the other hand, refers to relation detection in the scene graph. If the relation entity has a similar name to the entity in the answer and the relation content is consistent, it determines whether the relation recognition is accurate after judging the IOU between the entity and the answer.

[0166] The table shows the VILA-SFT-25K baseline experiment, which tests the scene graph generation capability after fine-tuning the scene graph instructions on the VILA-8B model; experiments starting with Qwen2-VL used Qwen2-VL-2B as the base model, and experiments starting with Qwen2-5-VL used Qwen2.5-VL-3B as the base model; SFT and grpo represent the model training method, ng16 indicates that the number generation parameter is set to 16 during grpo training, 10k and 25K represent the amount of training data, ep4 and 1000s represent the number of training steps (ep4 is 4 epochs, 1000s is 1000 steps), and -sg-qa indicates that the training task includes scene graph reasoning generation as well as scene graph reasoning question answering.

[0167] Experimental Analysis: Two tables focus on evaluating scene graph generation capabilities, using the VILA-SFT-25K training results as a benchmark. Detailed evaluations were conducted on the PSG test set, focusing on both entity and relation dimensions. Experimental data shows that the Qwen model, which simply employs scene graph fine-tuning (Qwen2-VL-SFT-10k) or reinforcement learning (Qwen2-VL-grpo-10k-ng16), performs slightly worse than the inference-based fine-tuning approach. It is noteworthy that in the final version of the inference question-answering model, although the scene graph generation performance declined to some extent, this design enabled the model to deeply explore the inference value of scene graphs, achieving a functional expansion from "graph generation" to "graph inference." This explains the functional trade-off behind the seemingly weakened scene graph generation capability.

[0168] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0169] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.< / answer> < / think> < / subsg> < / extraction> < / sg> < / content> < / summary> < / subsg> < / sg> < / content> < / summary> < / sg> < / content> < / summary> < / summary> < / summary> < / content>

Claims

1. A method for constructing a scene graph question-answering reasoning model based on reinforcement learning, characterized in that, Includes the following steps: Obtain the initial scene dataset, call the existing large model interface, and set prompt words to guide the existing large model to perform in-depth processing on the initial scene dataset to build the first batch of training sets; The first phase of reinforcement learning training is performed on the pre-built multimodal large model using the first batch of training set; Obtain the LLaVA-CoT dataset, filter it, and input it into the multimodal large model trained in the first stage to obtain the inference results; The existing large model interface is called to evaluate the accuracy of the inference results of the multimodal large model, and the errors in the inference results are corrected in a targeted manner to obtain the reconstructed dataset. A two-stage inference process is then performed on the reconstructed dataset to realize intelligent derivation from the generalized scene graph to the targeted answer, and the second batch of training set is obtained. The second batch of training set is used to conduct the second stage of reinforcement learning training on the multimodal large model. After the training is completed, the final scene graph question answering reasoning model is obtained.

2. The method for constructing a scene graph question-answering reasoning model based on reinforcement learning according to claim 1, characterized in that, The data structure for each data entry in the first batch of datasets is as follows: [Image, Detailed Image Description, Brief Image Summary, List of Image Entities and Relationships, Image Scene Graph].

3. The method for constructing a scene graph question-answering reasoning model based on reinforcement learning according to claim 1, characterized in that, The process of filtering the LLaVA-CoT dataset includes: Image type filtering: Select natural scene images suitable for scene graph representation, and remove charts, mathematical diagrams, structured images in the natural science field, and datasets whose main purpose is text recognition; Question and answer type filtering: Filter out question and answer data with clear accuracy assessment methods, and prioritize retaining multiple-choice questions and open-ended simple-response questions; After two rounds of selection, a subset of the LLaVA-CoT dataset was obtained. The data structure of each data point in the subset is as follows: [Image, User Question, Reasoning Process, Accurate Answer].

4. The method for constructing a scene graph question-answering reasoning model based on reinforcement learning according to claim 3, characterized in that, The process of the multimodal large model trained in the first stage performing inference on the selected LLaVA-CoT dataset includes: Adjust the generation parameters of the multimodal large model, and randomly output four inference results for each data point in the subset. The data structure of each data point in the subset is expanded to: [image, four image inference and scene graph results, user question, inference process, accurate answer].

5. The method for constructing a scene graph question-answering reasoning model based on reinforcement learning according to claim 1, characterized in that, The process of accuracy evaluation and correction of inference results for multimodal large models includes: Call the existing large model interface, set prompt words, and evaluate the accuracy of each part of the inference result of the multimodal large model for each data point; If all parts of the reasoning result of a certain data are determined to be accurate, then it is directly included in the final reconstructed dataset; If all four inference results corresponding to a certain data point contain errors, the number of correct parts in each inference result will be automatically calculated, and the inference result with the highest accuracy will be selected to enter the reconstruction process. During reconstruction, the existing large model interface will be called, and prompts will be set to guide the existing large model to make targeted corrections to the erroneous parts. After correction, the model will be included in the reconstructed dataset. The data structure for each data point in the reconstructed dataset is as follows: [Image, detailed description of the image, brief summary of the image, list of image entities and relationships, image scene diagram, user question, reasoning process, accurate answer].

6. The method for constructing a scene graph question-answering reasoning model based on reinforcement learning according to claim 1, characterized in that, The two-stage inference process performed on the reconstructed dataset includes a scene subgraph extraction inference process and a multimodal inference generation process; the scene subgraph extraction inference process includes: Call the existing large model, set prompt words, and semantically align user questions with entities and relationships in the scene graph; Extracting relevant entities and triples based on question keywords; Construct targeted scenario subgraphs to form a structured representation that is strongly related to the problem. The final output format is JSON, which contains the entities and relationships that the problem focuses on. The multimodal reasoning generation process includes: By utilizing the extracted scene subgraphs and original information, the existing large model performs multi-round reasoning generation: integrating multi-source information such as scene subgraphs, detailed descriptions, and user questions; A thought chain reasoning strategy is employed to generate intermediate reasoning steps; Apply constraint satisfaction mechanisms to ensure logical consistency; The final output includes an answer text and explanation process that conforms to academic standards.

7. The method for constructing a scene graph question-answering reasoning model based on reinforcement learning according to claim 1, characterized in that, After calling the existing large model interface to perform a two-stage inference process on the reconstructed dataset, it also includes: A dual verification mechanism is used to verify the generated final answer. For multiple-choice questions, the accuracy is judged directly by matching the options. For open question-and-answer type data, in addition to basic rule judgment, the semantic understanding capabilities of the existing large model are also introduced for secondary verification; If the final verification result shows that the answer is inaccurate, the retry mechanism will be automatically triggered to re-call the existing large model interface to regenerate the data until the accurate answer is obtained or the preset retry limit is reached. The final data structure for each item in the second batch of training sets is as follows: [Image, detailed image description, brief image summary, list of image entities and relationships, image scene graph, scene graph extraction and reasoning, scene subgraph, reasoning, user question, accurate answer].

8. The method for constructing a scene graph question-answering reasoning model based on reinforcement learning according to claim 1, characterized in that, In the first phase of reinforcement learning training, a format reward function and an accuracy reward function are constructed. The format reward function constrains the output of the multimodal large model from three dimensions: generation step format reward, length constraint reward, and scene graph format reward, so that the generated scene graph conforms to the predetermined format standard. The accuracy reward function constrains the output of the multimodal large model from two dimensions: scene graph relationship reward and scene graph entity reward, focusing on the accuracy of entity and relationship recognition in the scene graph.

9. The method for constructing a scene graph question-answering reasoning model based on reinforcement learning according to claim 8, characterized in that, In the second phase of reinforcement learning training, a multi-type data hybrid training strategy is adopted, covering scene graph generation inference data, scene graph open question answering inference data, and scene graph multiple-choice question answering inference data. For training the inference data generated from the scene graph, the evaluation mechanism of the scene graph accuracy reward function from the first stage is used. For training on open question-and-answer reasoning data or multiple-choice question-and-answer reasoning data in scene graphs, an absolute content matching combined with rule-based judgment is used. By accurately comparing the answer content with the established rules, the validity of the multimodal large model output is determined.

10. The method for constructing a scene graph question-answering reasoning model based on reinforcement learning according to claim 1, characterized in that, The existing large model interface being called is either the GPT-4o large model or the Gemini large model.

Images