Intelligent SQL query statement generation method and system based on multiple modes

Abstract

The invention discloses an intelligent SQL query statement generation method and system based on multiple modes, and belongs to the technical field of artificial intelligence. In order to solve the problems that an existing Text-to-SQL method cannot process visual dependency query, lacks a multi-modal reference data set, is difficult in mapping of visual information and a database mode and does not fully utilize visual-schema alignment, the method comprises the following steps: constructing a multi-modal training sample containing a natural language rewriting problem, a target image and a database schema; a three-layer progressive vision-schema link strategy is adopted, supervision fine tuning and reinforcement learning staged training are combined, and a composite reward function containing a vision-schema alignment reward is introduced in reinforcement training for optimization. According to the method, the executable SQL statement can be accurately generated by the model under the multi-modal input condition, and effective association between the visual information and the database mode is realized.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method and system for intelligent generation of SQL query statements based on multimodality. Background Technology

[0002] Understanding this invention requires reliance on two core, well-known technologies: Text-to-SQL (TTL) technology and Visual-Language Models (VLMs). Text-to-SQL technology is the mainstream method for implementing natural language interfaces for databases. Its core principle is to transform unstructured natural language questions input by users into structured SQL query statements that can be executed in relational databases. This technology relies on the natural language processing model's deep analysis of the question's semantics and precise matching of the database schema to establish a correspondence between natural language semantics and SQL syntax rules and schema components (tables, columns, and values). In recent years, with the development of Large Language Models (LLMs), Text-to-SQL technology has been able to handle complex scenarios involving dirty data (such as non-standard field values ​​and redundant text information) and multi-step reasoning requirements by introducing methods such as thought chain hints (guiding the model to reason step-by-step about the association logic between semantics and schema), in-context learning (allowing the model to quickly master the task logic through a few examples), and specialized fine-tuning (optimizing model parameters for SQL generation tasks). The accompanying evaluation dataset has also evolved from the Spider dataset, which was mainly based on clear structure, to the BIRD dataset, which is closer to industrial scenarios. This dataset not only covers large-scale real databases, but also introduces data noise and multi-step inference tasks, thereby improving the evaluation criteria for model capabilities.

[0003] The core capability of visual-language model technology lies in achieving cross-modal understanding and reasoning of visual and linguistic information. Its technical principle involves converting images into high-dimensional visual feature vectors using an image encoder, converting natural language into text feature vectors using a text encoder, and then using attention mechanisms or feature fusion layers to achieve interaction and association between the two types of features, thereby supporting tasks such as image description and visual question answering. Mainstream models include CLIP, InternVL3-14B, and Gemini 2.5Pro. These models can initially parse entities (such as people and products), attributes (such as color and shape), and scene information (such as meetings and rainy scenes) in images, but they have not yet established a deep association with structured database schemas and cannot directly convert visual information into the tables, columns, and values ​​required for structured queries.

[0004] In existing technologies, the solutions closest to the objectives of this invention fall into two categories: one is a plain text-to-SQL solution based on LLMs, and the other is a VLMs solution with cross-modal understanding capabilities but without schema alignment. The plain text-to-SQL solution based on LLMs is the most widely used in industry. Its implementation process includes: obtaining the user's input natural language question and the target database schema (including table names, column names, field types, and inter-table relationships); concatenating the natural language question and schema information into a fixed-format input sequence, inputting it into an LLM (such as Llama-3-7B or GPT-4) specifically fine-tuned for the Text-to-SQL task; the model identifies the query intent through a semantic parsing module, associates the intent with specific tables and columns through a schema matching module, and generates a query statement conforming to SQL syntax rules; finally, the execution function verifies the correctness of the SQL syntax and execution results, and if the results do not match expectations, the model is returned for optimization. This solution is limited to supporting only plain text input and cannot handle scenarios that rely on visual information to convey query elements.

[0005] Another related technical solution is a multimodal question answering scheme combining Visual Models (VLMs). The process involves acquiring the user's natural language question and a set of related images, parsing the image content using a VLM image encoder to extract entity, attribute, and scene information, parsing the question's semantics using a text encoder, and combining the two types of information to generate a natural language answer. Some schemes convert the answer into structured data, but without establishing a mapping with the database schema, they cannot generate executable SQL statements. For example, in existing multimodal datasets (such as Flickr30K and COCO question answering datasets), visual content is only used as supplementary information, not a necessary condition for generating query results. The model can directly deduce the answer from the text information without actual cross-modal reasoning. Furthermore, these schemes lack an alignment mechanism between visual information and schema components, failing to map image features to specific columns or values ​​in the database, thus hindering multimodal-driven SQL generation.

[0006] From the perspective of technical mechanisms and practical application effects, existing technologies suffer from three core defects, all of which directly limit convenient access to structured databases in multimodal scenarios. First, plain text (Text-to-SQL) technology has a unimodal limitation; its architecture only supports text from the input to the output layer and cannot parse visual information, causing it to fail in scenarios requiring combined visual evidence. For example, when users request queries such as "querying the quarterly sales of a brand in the image," "corresponding to yield data based on the state of farmland vegetation in satellite images," or "identifying abnormal cells in microscopic images and querying case information," the core query elements are conveyed through visual content. Plain text (Text-to-SQL) cannot obtain this information, leading to SQL generation failure and limiting its application to plain text scenarios. Secondly, while VLMs technology possesses cross-modal understanding capabilities, it faces significant obstacles in aligning visual content with the database schema: There is a lack of dedicated benchmark datasets for multimodal SQL generation; existing datasets often contain visual content that is not essential or lacks deep association with the schema, meaning models may rely solely on text shortcuts to complete tasks, failing to effectively test cross-modal reasoning capabilities; the unstructured nature of visual data inherently differs from the symbolic and structured nature of database schemas, making it difficult for models to accurately map image entities, attributes, and context to tables, columns, or values, thus hindering the transformation of visual information into structured queries; existing training paradigms only optimize traditional SQL generation metrics without designing specific training objectives and reward mechanisms for visual-schema alignment, resulting in low cross-modal SQL generation accuracy and poor executability, making it difficult to meet industrial needs. Finally, from the perspective of technical parameters and engineering efficiency, existing methods suffer from high data annotation costs and slow training convergence: traditional supervised learning requires large-scale annotation of multimodal samples of "natural language problems + images + schema + SQL", and the annotation requires both SQL knowledge and visual understanding ability, which is costly; reinforcement learning lacks effective intermediate rewards and only relies on binary rewards. The sparse rewards lead to slow training convergence, and usually thousands of training steps are needed to achieve basic performance, making it difficult to apply efficiently to engineering scenarios. Summary of the Invention

[0007] The present invention aims to solve the following technical problems.

[0008] (1) Existing Text-to-SQL technology only supports plain text input and cannot handle query requirements that rely on visual evidence.

[0009] (2) In the field of multimodal SQL generation, there is a lack of dedicated benchmark datasets that can effectively test the collaborative reasoning ability between vision and database schema. Models can easily complete tasks through text shortcuts.

[0010] (3) There is a natural mapping gap between visual unstructured information and database schema structured information, making it difficult for the model to map entities and attributes in the image to schema components.

[0011] (4) The existing training paradigm does not have a specific target for vision and schema alignment. The model cannot actively learn cross-modal grounding capabilities, resulting in low accuracy and poor executability of SQL generation in complex scenarios.

[0012] The present invention addresses the above-mentioned technical problems by adopting the following technical solution.

[0013] A method for intelligently generating SQL query statements based on multimodality includes the following steps: 1) Obtain natural language questions, database patterns, and image information; parse the natural language questions and database patterns; extract visual content from the image information; and generate training samples containing natural language questions, image information, database patterns, and real query statements based on the parsing results. 2) The multimodal model is trained in stages based on the generated training samples, including two stages: supervised fine-tuning and reinforcement learning training, so that the model can generate predictive query statements based on natural language questions, image information and database patterns. 3) Input the natural language question, image information, and database pattern into the trained multimodal model to generate a predicted query statement; 4) Execute and verify the generated predictive query statements, including execution results, structure matching, and field matching verification, and determine the valid predictive query statements based on the verification results; 5) Execute valid predictive query statements in the database and convert the resulting structured query results into natural language descriptions.

[0014] Furthermore, the generation of training samples in step 1) includes: parsing real query statements, extracting the database tables, fields and values ​​involved, and generating candidate visual entities in conjunction with natural language questions.

[0015] Further, the generation of training samples in step 1) includes: obtaining candidate images related to candidate visual entities from external image sources, and filtering the candidate images to obtain image information.

[0016] Furthermore, step 1) of filtering candidate images includes: The first layer of association is determined based on the semantic correspondence between image content and specific values ​​in the database; The second-level association is determined based on the semantic correspondence between image visual attributes and database field names; The third layer of association is determined based on the semantic correspondence between image scene content and abstract concepts in the database; Based on the combined scores of the three layers of correlation, the image with the highest score is selected as the target image.

[0017] Furthermore, in step 1), when parsing the natural language question, the explicit visual entities in the question are replaced based on the image information to obtain a natural language question that includes image orientation.

[0018] Furthermore, step 2) of supervised fine-tuning of the multimodal model includes: Encode the target images of the training samples into visual features; The aforementioned visual features are fused with the text features of the rewriting problem and database pattern through an attention mechanism to obtain fused features; The fused features are used as input to train the multimodal model, with the goal of generating inference trajectories and query statements. The model parameters are iteratively optimized using an autoregressive cross-entropy loss function.

[0019] Furthermore, step 2) of training the multimodal model using reinforcement learning includes: Multiple samples are taken from the natural language questions, target images, and database patterns of the training samples to generate multiple candidate query statements; For each candidate query statement, a composite reward is calculated, including three dimensions: basic correctness, intermediate similarity, and pattern alignment. The advantage value of the candidate query statement relative to the average reward within the group is also calculated. Based on the advantage value, the objective function of the optimization algorithm for grouping relative strategy is updated to update the model output strategy. This process is repeated iteratively until the model converges. When the model fails to meet the performance index, the candidate samples are returned to the reinforcement training phase to continue optimizing the model parameters.

[0020] Furthermore, the verification of the predicted query statement in step 4) includes: calculating three indicators: execution accuracy, structure matching rate, and field matching rate, and judging whether the predicted query statement is effective based on the indicator results; when any indicator fails to reach the set threshold, the predicted query statement is returned to the reinforcement learning training stage in step 2) for model optimization.

[0021] Furthermore, the generation of natural language description in step 5) includes: inputting the obtained structured query results into the language model, and having the language model generate a natural language description based on the database query results.

[0022] A multimodal SQL query statement intelligent generation system includes: The dataset construction module is used to acquire natural language questions, database schemas, and real query statements. It parses the natural language questions and database schemas, extracts the relevant database tables, fields, and values ​​from the real query statements, generates candidate visual entities based on the natural language questions and extracted content, obtains candidate images related to the candidate visual entities from external image sources, performs a three-level association evaluation on the candidate images and selects the image with the highest score as the target image, and replaces the explicit visual entities in the natural language questions based on the target image to form training samples containing image orientation. The model training module is used to train the multimodal large model in stages based on the training samples generated by the dataset building module. This includes a supervised fine-tuning stage, which enables the model to generate outputs consistent with real query statements based on natural language questions, target images, and database patterns, and a reinforcement learning stage, which updates the model generation strategy through multi-candidate SQL sampling, compound reward calculation, and policy optimization, iterating until the model converges. The SQL generation and verification module receives natural language questions, target images, and database schemas, parses the structural information of the natural language questions and database schemas, generates predictive query statements, and performs execution verification, structure matching verification, and field matching verification on the generated query statements. Based on the verification results, it determines the valid predictive query statements. The results feedback module is used to feed back the verification results of the SQL generation verification module to the model training module. When the predicted query statement does not meet the verification indicators, it is used to adjust the model generation strategy in the subsequent reinforcement training stage.

[0023] The present invention mainly employs the following targeted technical means.

[0024] To address problem (1), this invention constructs a multimodal sample combining "rewriting natural language problems, target images and database schemas" and combines it with a visual-schema alignment strategy to support visually dependent queries, enabling the model to extract key query elements from visual input.

[0025] To address problem (2), this invention constructs the MM2SQL-Bench dataset and adopts a system process of "basic data filtering → visual entity mining → image filtering → three-layer visual-schema linking → problem rewriting" to achieve deep and multi-dimensional association between visual information and database schema, ensuring that the model must generate correct SQL through real cross-modal reasoning, thus filling the gap in dedicated benchmarks in this field.

[0026] To address problem (3), this invention designs a three-layer progressive visual-schema linking strategy, including direct entity grounding (associating image entities with database values), attribute-based instantiation (associating visual details with database columns), and context and reasoning association (associating scene information with abstract concepts). Furthermore, a visual-schema alignment reward is introduced during training to guide the model to learn the mapping rules from visual information to schema components, thus overcoming the natural mapping gap.

[0027] To address problem (4), this invention employs a phased training framework of "Supervised Fine-tuning (SFT) and Reinforcement Learning (RL)". In the SFT phase, a warm-up dataset is used to instill basic multimodal reasoning capabilities. In the RL phase, a composite reward function (integrating basic correctness, intermediate similarity, and schema alignment rewards) is used to optimize the model, simultaneously mitigating reward sparsity and improving training convergence speed, as well as the accuracy and executability of generated SQL. All four technical problems revolve around the core objective of "Visual-schema alignment-driven SQL generation under multimodal input," are interconnected, and can be effectively solved through the technical solution of this invention.

[0028] The present invention has achieved the following beneficial effects.

[0029] 1. This invention constructs a visual evaluation sample set with "degraded-high-quality" pairings, solving the problem of the lack of accurate optimization references in existing technologies. By screening defective samples generated by the existing Text2Vis method and high-quality samples from the benchmark dataset, and forming a sample set by linking "natural language questions - tabular data" one by one, it provides a dual reference of defects and targets for visual knowledge extraction, avoiding blindness in knowledge extraction.

[0030] 2. This invention proposes a visual knowledge extraction framework of "layered comparison prompts and multi-granularity quality assessment," addressing the problems of inaccurate knowledge extraction and lack of quality screening in existing technologies. Layered prompts guide VLMs to extract knowledge from three levels: task understanding, visual features, and code improvement. Multi-granularity assessment ensures the accuracy and usability of knowledge samples through a generator to generate multiple paths, a validator to verify validity, and a discriminator to score and screen, overcoming the limitations of traditional knowledge extraction methods that rely on "single generation without screening."

[0031] 3. This invention designs a two-stage VLM fine-tuning strategy of "Supervised Fine-tuning (SFT) and Iterative Preference Alignment (DPO)" to address the lack of professional Text2Vis optimization capabilities in existing VLMs. The SFT stage enables the model to acquire basic defect identification and code improvement capabilities, while the DPO stage constructs a preference dataset through the VC field correctness reward function to gradually align with user intent. Compared with a single fine-tuning strategy, this significantly improves the accuracy and user matching of model improvement suggestions.

[0032] 4. This invention employs a plug-in closed-loop optimization design to address the issues of poor compatibility and lack of feedback loops in existing technologies and other Text2Vis methods. No modification to the existing method's generation logic is required; only the generated chart and VC need to be input to call the finely tuned VLMs for optimization. Simultaneously, by combining natural language interpretation and knowledge base data association, a complete "generation-evaluation-improvement-feedback" closed loop is formed, overcoming the limitation of traditional methods where "generation is the endpoint."

[0033] 5. This invention introduces a new mechanism linking visual feedback with a knowledge base, addressing the issue of insufficient user understanding in existing technologies. Optimized charts and defect improvement descriptions are integrated into natural language feedback, and the knowledge base can be invoked to execute corresponding VC queries to output raw data results. This helps non-technical users understand the logic behind the visual improvements and the data sources, enhancing the practicality of the technology. Attached Figure Description

[0034] Figure 1 This is a flowchart of the construction process of the MM2SQL-Bench benchmark dataset.

[0035] Figure 2 This is an overview of the reward indicators. Detailed Implementation

[0036] To make the various technical features, advantages, or effects of the present invention more apparent and understandable, detailed descriptions are provided below through embodiments.

[0037] An embodiment of the present invention provides a method for intelligently generating SQL query statements based on multimodality, comprising the following steps: Step 1: Construct the MM2SQL-Bench benchmark dataset, such as Figure 1 As shown.

[0038] (1) Acquisition of basic data.

[0039] Using the BIRD dataset as the initial data source, the corresponding database is connected via Python's SQLAlchemy library. The actual SQL query Y is executed for each sample, and only samples that are successfully executed and whose results match the annotations are retained. Where Q represents the natural language problem, S represents the database schema, and Y represents ground-truth SQL, forming the basic dataset.

[0040] Alternatively, datasets such as Spider and WikiSQL, which contain natural language questions, database schemas, and real SQL triple structures, can be used to replace BIRD. In implementing this replacement, a complete database environment needs to be established, ensuring that the SQL queries for all samples can be validated using SQLAlchemy. Simultaneously, entity distribution characteristics need to be re-analyzed, samples containing visual entities (such as product appearance or organization logos) should be selected, and entity mining prompt templates should be adjusted. Finally, multi-model consistency checks and manual review are required to ensure that the visual-schema association quality maintains the original standard. This type of replacement only involves the data source level and does not change the core architecture of the three-layer visual-schema link.

[0041] (2) Visual entity mining.

[0042] The sqlparse tool is used to parse the Y of each base sample and extract its minimum schema context: in For a set of tables, For a set of columns, It is a set of values.

[0043] Calling a large language model (such as GPT-4) as an entity miner, combined with Q and Generate a list of candidate visual entities ,make sure The entities in the middle satisfy the requirements of being visually recognizable (such as company logos and product appearances) and being relevant to the core query.

[0044] (3) Acquisition of relevant images.

[0045] via Google Images API The entity phrase is used as a keyword to retrieve 100 to 200 candidate images. Then, the CLIP-ViT-L / 14 model is used to calculate the cosine similarity between each image and the entity text phrase, and a threshold is set. Images with similarity higher than a threshold are retained to remove noise, resulting in a set of semantically related candidate images.

[0046] Models such as InternVL2-4B or EVA-CLIP can be used to replace CLIP-ViT-L / 14. In implementing this replacement, feature dimension transformation is required. For example, since InternVL2-4B outputs 1024-dimensional features, the feature fusion layer weights need to be retrained, and the feature alignment effect needs to be verified. The similarity threshold should be recalibrated using grid search. When evaluating computational efficiency, a hierarchical filtering mechanism can be introduced, using a lightweight model for initial screening followed by fine-tuning. Throughout the process, the accuracy of the visual-semantic association must be maintained, and the quality of the filtered images needs to be verified using a cosine similarity distribution curve.

[0047] (4) Visual-pattern link alignment.

[0048] This is the core step in the construction process, aiming to establish a deep connection between image content and database schema. A three-tiered progressive linking strategy is used to score and filter candidate images.

[0049] First-level direct entity link: Evaluate whether the image can be directly mapped to a specific value in the database (e.g., the company logo in the image corresponds to the company name in the database).

[0050] The second layer is attribute-based instantiation: evaluating whether the image's visual attributes (such as color and shape) can be associated with a specific column in the database (such as "red" corresponding to the color column in the product table).

[0051] The third layer of context and inference association: This assesses whether the image scene context (such as weather or event type) can be inferred to be related to abstract concepts in the database (e.g., rainy day is associated with the rainfall column in the weather data table, as shown in the image). Following this three-layer strategy, InternVL3-14B is used to score candidate images, and the total image score is calculated using the following weighted formula: in For the first Binary scores for layer links. The image with the highest score is selected as the target image.

[0052] (5) Rewrite the collaborative problem.

[0053] Calling Llama-3-70B to replace explicit visual entities in Q with indicative references (e.g., replacing "red product" with "the red product shown in the image") yields the rewrite problem. This ultimately forms a quadruple sample. The MM2SQL-Bench benchmark dataset consists of 5297 samples, which are divided into easy, medium, and hard levels according to SQL complexity and inference level.

[0054] Step 2: Training the multimodal SQL generation model.

[0055] This phase employs a phased training framework combining supervised fine-tuning (SFT) and reinforcement learning (RL), with reward metrics such as... Figure 2 As shown.

[0056] (1) Training data preparation.

[0057] 1) Divide 4020 samples from the MM2SQL-Bench benchmark dataset as a supervised fine-tuning dataset. The remaining 1277 samples were used as the evaluation set.

[0058] 2) Call the teacher model Gemini 2.5 Pro for The sample generates an inference trajectory T and a prediction SQL A, where the inference trajectory T describes the steps from the multimodal input to the generation of the schema component, and the prediction SQL A is the output of the teacher model.

[0059] 3) Define the execution function By comparing teacher models, the execution results of SQL can be predicted. Execution results of the actual SQL The equivalence will Divided into two parts: Inference Warm-up Dataset :satisfy The sample format is ; Challenging datasets :satisfy The sample format is ; in," " indicates equivalence, " "Indicates that they are not equivalent"

[0060] (2) Supervisory fine-tuning (SFT).

[0061] 1) Select Llama-3-7B as the initial model. , target image The visual features, encoded as 768-dimensional using CLIP, are fused with the text features of the rewrite problem Q' and the database pattern S through an attention mechanism and then input into the model.

[0062] 2) With the goal of generating the inference trajectory T and the output SQL Y, the autoregressive cross-entropy loss function is adopted: in, The loss function during the supervised fine-tuning phase depends on the model parameters. The smaller the value, the more accurate the model prediction; This indicates the inference warm-up dataset. Calculate the expectation from the sampled samples; The model input includes the rewrite problem, the target image, and the database schema; This represents the target output sequence, including the inference trajectory T and the actual SQL Y; Represents a sequence Total length (number of lexical units); Represents a sequence At time step Lexical units (such as words or tags); Represents a sequence At time step All previous morphemes; Indicates the model in parameters Given the historical context and input Predict the current word element The conditional probability.

[0063] 3) The AdamW optimizer was used for training, with a learning rate of 2e-5 and a batch size of 8. After training for 10 epochs, the warm-up model Mwm was obtained.

[0064] (3) Reinforcement learning (RL).

[0065] 1) with The strategy is the reference strategy. The Grouped Relative Policy Optimization (GRPO) algorithm is used for optimization. GRPO provides dense guidance through a composite reward function and gradually adjusts the model parameters.

[0066] 2) Detailed GRPO optimization process: a. Grouped sampling candidate SQL: For each input sample (From challenging datasets) Sampling), from the reference strategy 14 candidate query SQL samples were selected. Explore the strategy space through diversity.

[0067] b. Calculate the compound reward value: for each candidate SQL ， Calling the compound reward function Calculate rewards The composite reward function integrates multi-dimensional signals from three levels: basic correctness, intermediate similarity, and pattern alignment.

[0068] c. Calculate the advantage value: the advantage value of each candidate SQL statement. ,in This is the average reward within the group. A positive dominance value indicates that the candidate SQL is better than the group average, and the strategy is guided to update in its direction.

[0069] d. Policy update objective function: Adjust the model parameters θ by optimizing the GRPO objective function. The objective function is: in, The probability of generating candidate SQL for the current strategy. The reference policy probability is used. Gradient ascent is employed to maximize this objective, with each update based on... Batch data from sampling.

[0070] e. Iteration and convergence: Repeat the above steps for iterative training, processing one batch of samples in each step, and train until the model performance is stable after 600 steps, obtaining the final model MM2SQL-7B.

[0071] 3) Composite reward function design: in It covers three categories of rewards: basic correctness, intermediate similarity, and pattern alignment; The weight hyperparameters are optimized using grid search, for example... Prioritize visual-schema alignment.

[0072] Reward component description: Basic accuracy bonus: Execution accuracy Syntax execution reward Format rewards .

[0073] Intermediate similarity reward: SQL similarity Based on the longest common substring normalized to 0-1; result similarity Conditions Execution successful.

[0074] Pattern alignment bonus: Pattern link Visual schema alignment The F1 score of the predicted SQL and the real SQL in the visual dependency section was evaluated.

[0075] The model parameters are optimized using the GRPO objective function described above, and the model is trained for 600 steps until convergence, resulting in the final multimodal SQL generation model MM2SQL-7B.

[0076] Step 3: SQL generation and verification.

[0077] Aligned multimodal input samples Input the trained MM2SQL-7B model to generate a predicted query. Then, the execution accuracy (Exec Acc), structural matching rate (Struct Match), and column matching rate (Col Match) are calculated. If Exec Acc ≥ 35%, Struct Match ≥ 30%, and Col Match ≥ 45%, the predicted SQL is considered effective; if any of the indicators does not reach the threshold, the process returns to the reinforcement learning stage in step 2 to continue optimizing the model parameters.

[0078] Step 4: Natural language result generation.

[0079] Execute effective commands in the target database. The system retrieves structured query results, calls Llama-3-7B to convert the results into natural language descriptions, such as stating "the inventory of the red product shown in the figure is 50 pieces", and returns it to the user to complete the entire processing flow.

[0080] Another embodiment of the present invention provides a multimodal SQL query statement intelligent generation system, comprising: The dataset construction module is used to acquire natural language questions, database schemas, and real query statements. It parses the natural language questions and database schemas, extracts the relevant database tables, fields, and values ​​from the real query statements, generates candidate visual entities based on the natural language questions and extracted content, obtains candidate images related to the candidate visual entities from external image sources, performs a three-level association evaluation on the candidate images and selects the image with the highest score as the target image, and replaces the explicit visual entities in the natural language questions based on the target image to form training samples containing image orientation. The model training module is used to train the multimodal large model in stages based on the training samples generated by the dataset building module. This includes a supervised fine-tuning stage, which enables the model to generate outputs consistent with real query statements based on natural language questions, target images, and database patterns, and a reinforcement learning stage, which updates the model generation strategy through multi-candidate SQL sampling, compound reward calculation, and policy optimization, iterating until the model converges. The SQL generation and verification module receives natural language questions, target images, and database schemas, parses the structural information of the natural language questions and database schemas, generates predictive query statements, and performs execution verification, structure matching verification, and field matching verification on the generated query statements. Based on the verification results, it determines the valid predictive query statements. The results feedback module is used to feed back the verification results of the SQL generation verification module to the model training module. When the predicted query statement does not meet the verification indicators, it is used to adjust the model generation strategy in the subsequent reinforcement training stage.

[0081] Another embodiment of the present invention provides a computer device including a memory and a processor, wherein a computer program stored in the memory includes instructions for performing the above-described steps; another embodiment provides a computer-readable storage medium in which a stored computer program performs the above-described steps.

[0082] Specific examples.

[0083] Taking a user's input query for the inventory of the red product shown in the image as an example, the first step is to construct multimodal samples according to step 1 above. In the basic data acquisition phase, samples containing natural language questions are filtered from the BIRD e-commerce subset. Database schema and real SQL The sample; after verifying that Y can be executed correctly in the database, the triple is included in the base dataset.

[0084] Subsequently, in the visual entity mining stage, Y is analyzed and the minimal schema context is extracted. And generate a list of visual entities based on this context. In the relevant image acquisition and filtering process, with 100 candidate images were retrieved for the keyword. After CLIP calculation, 20 images with a similarity greater than or equal to 0.7 were retained as the candidate set. Then, the candidate images were scored layer by layer according to a visual-schema linking strategy; for example, the image of the red shirt was scored... , , The image with the highest weighted score of 0.5 was identified as the target image. Based on this, the original problem is rewritten as follows: To form a complete sample .

[0085] In step 2 above, the sample is assigned to the supervised fine-tuning dataset. Reasoning trajectories are generated through teacher models. And predicting SQL After verifying that the execution results of Exec(A) and Exec(Y) are consistent, the sample is included in the inference warm-up dataset. Subsequently, supervised fine-tuning training was performed to obtain... Furthermore, during the reinforcement learning phase, a compound reward function was used to further optimize the algorithm for 600 steps, ultimately resulting in MM2SQL-7B.

[0086] In the final reasoning stage, Enter MM2SQL-7B to generate The prediction SQL's execution accuracy, structure matching rate, and column matching rate all reached the set thresholds, thus being deemed valid. After execution in the target database, the inventory value of 50 was obtained, which was then converted into natural language by Llama-3-7B and fed back to the user.

[0087] Test results on the MM2SQL-Bench evaluation sample show that MM2SQL-7B has an execution accuracy of 36.73%, a structure matching rate of 37.20%, and a column matching rate of 48.36%, which are significantly improved compared with open source models and some proprietary models, indicating that the data construction process and model training framework have good performance.

[0088] The beneficial effects achieved by this invention are explained.

[0089] (1) Systematic construction of the MM2SQL-Bench benchmark dataset.

[0090] This invention constructs an MM2SQL-Bench dataset containing 5297 samples through a closed-loop process consisting of basic data screening, visual entity mining, image semantic filtering, three-layer visual-schema linking, and collaborative problem rewriting. Core technical methods include: ensuring data authenticity based on executable SQL samples; utilizing GPT-4 to mine entities that simultaneously possess visual representativeness and core query relevance to ensure the necessity of visual information; and employing CLIP filtering (threshold filtering). Remove image noise; design a three-layer progressive link structure of direct entity grounding, attribute instantiation, and context association to establish a deep association between visual information and schema; and force cross-modal reasoning through a problem rewriting mechanism.

[0091] The beneficial effects of the aforementioned techniques are mainly reflected in the following aspects: First, they fill the gap in professional benchmark datasets for multimodal SQL generation. In existing datasets (such as Flickr30K-VQA and COCO-QA), visual content is only used as auxiliary information, and models can complete tasks through text shortcuts. MM2SQL-Bench performs consistency checks through three visual language models and combines human experts' review of the rationality of visual-schema associations to ensure that visual information is a necessary condition for generating correct SQL, thereby providing a highly realistic and standardized carrier for model evaluation and promoting the standardization of the field. Second, based on SQL complexity (including JOIN, subqueries, aggregate functions, etc.) and inference level, samples are divided into different difficulty levels, which can be used to accurately evaluate the model's cross-modal capabilities under different inference complexities, providing a clear direction for subsequent technology iterations.

[0092] (2) A phased supervision and fine-tuning and reinforcement learning training framework.

[0093] This invention proposes a phased training framework consisting of a supervised fine-tuning (SFT) phase and a reinforcement learning (RL) phase. The core technical means include: the SFT phase uses an inference warm-up dataset. (Samples generated by the teacher model with high accuracy) are used as training data, and the autoregressive cross-entropy loss function is applied. The fundamental mapping relationship from multimodal input to inference trajectory and then to SQL generation is learned; the RL phase employs a reward system that includes visual-schema alignment. The composite reward function is optimized using the GRPO algorithm to prioritize enhancing cross-modal grounding capabilities.

[0094] The framework demonstrated significant advantages in experiments: First, the final generated MM2SQL-7B model achieved an execution accuracy of 36.73% on the MM2SQL-Bench evaluation set, significantly higher than the open-source model (Llama-4Maverick at 27.92%, Qwen2.5-VL-32B at 26.27%) and the SQL expert model (XiYanSQL-7B at 24.82%). It also surpassed the leading proprietary model Gemini 2.5Pro (29.72% and 46.08%, respectively) in result structure matching rate (StructMatch = 37.20%) and result column matching rate (ColMatch = 48.36%), effectively narrowing the performance gap between the open-source model and the proprietary model. Secondly, the composite reward function consists of a basic correctness reward (in binary form) and an intermediate similarity reward (ranging from 0 to 1), which effectively alleviates the sparsity of rewards during reinforcement learning training. This reduces the number of training convergence steps from approximately 850 steps in the traditional scheme to approximately 600 steps, thereby improving training efficiency while enhancing model performance.

[0095] (3) Three-layer progressive visual-schema linking strategy.

[0096] This invention addresses the difference between the unstructured attributes of visual information and the structured attributes of the schema, designing a three-layer visual-schema linking strategy consisting of direct entity grounding, attribute instantiation, and contextual reasoning association. The core technical approach involves using InternVL3-14B to score the association ability between images and schemas at each layer, and then applying a weighted formula... , , ( , , The weights are assigned to three levels: direct entity grounding, attribute instantiation, and context association. Image samples with higher relevance and requiring advanced inference are selected to enhance the model's learning of the mapping from multi-granular visual information to schema components.

[0097] The beneficial effects of this strategy include: First, it theoretically overcomes the limitation of existing multimodal models that can only describe visual content and are difficult to associate with structured schemas. It enables entities, attributes, and scenes in images to be accurately mapped to corresponding table and column entries in SQL queries, achieving cross-modal associations such as mapping a logo in an image to a company name field, a product appearance to a product shape field, and a rainy scene to a rainfall field. Second, experimental results show that MM2SQL-7B achieves an execution accuracy of 28.15% on difficult samples requiring contextual reasoning, significantly higher than Qwen2.5-VL-32B's 16.33%, demonstrating that this strategy effectively improves the model's ability to handle complex visual-schema associations.

[0098] (4) The idea of ​​training with few samples based on the teacher model.

[0099] In the training data preparation stage, this invention uses Gemini 2.5 Pro as the teacher model to supervise the fine-tuning of the dataset. Generate inference trajectory T and predicted SQL A, and classify them solely based on whether the execution results are consistent. and This eliminates the need for manually labeled multimodal sample reasoning logic.

[0100] This technical solution has the following advantages: Compared with manually annotating the mapping relationship between visual entities and schema components (which requires a lot of manual labor and professional SQL knowledge), this solution can reduce the data annotation cost by more than 60%; at the same time, the inference trajectory T provides the model with a clear cross-modal inference process, enabling MM2SQL-7B to achieve the basic performance of a traditional model with "more than 10,000 manually annotated samples" with only 4,020 SFT samples, which greatly reduces the data preparation threshold for the industrial application of the model.

[0101] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.

Claims

1. A method for intelligent generation of a multi-modal based SQL query statement, characterized in that, The method comprises the following steps: 1) obtaining a natural language question, a database schema and image information, parsing the natural language question and the database schema, and extracting visual content from the image information, and generating a training sample comprising the natural language question, the image information, the database schema and a real query statement based on the parsing result; 2) training the multi-modal model based on the generated training sample in stages, including supervised fine-tuning and reinforcement learning training two stages, so that the model can generate a predicted query statement according to the natural language question, the image information and the database schema; 3) inputting the natural language question, the image information and the database schema into the trained multi-modal model to generate a predicted query statement; 4) performing verification on the generated predicted query statement, including execution result, structure matching and field matching verification, and determining an effective predicted query statement based on the verification result; 5) executing the effective predicted query statement in the database, and converting the obtained structured query result into a natural language description.

2. The method of claim 1, wherein, The generation of the training sample in step 1) includes: parsing the real query statement, extracting the database table, field and value involved, and generating a candidate visual entity in combination with the natural language question.

3. The method of claim 2, wherein, The generation of the training sample in step 1) includes: obtaining a candidate image related to the candidate visual entity from an external image source, and screening the candidate image to obtain image information.

4. The method of claim 3, wherein, The screening of the candidate image in step 1) includes: determining a first layer association based on the semantic correspondence between the image content and the specific value of the database; determining a second layer association based on the semantic correspondence between the image visual attribute and the database field name; determining a third layer association based on the semantic correspondence between the image scene content and the database abstract concept; comprehensive three layer association score, select the highest score image as the target image.

5. The method of claim 1, wherein, When parsing the natural language question in step 1), the explicit visual entity in the question is replaced based on the image information to obtain a natural language question containing image direction.

6. The method of claim 1, wherein, The supervised fine-tuning of the multi-modal model in step 2) includes: encoding the target image of the training sample into visual features; fusing the above visual features and the text features of the rewritten question and the database schema through an attention mechanism to obtain fusion features; providing the fusion features as input to the multi-modal model for training, aiming to generate reasoning tracks and query statements, and using a self-attention cross-entropy loss function to iteratively optimize model parameters.

7. The method of claim 1, wherein, The reinforcement learning training of the multi-modal model in step 2) includes: sampling the natural language question, the target image and the database schema of the training sample multiple times to generate multiple candidate query statements; calculating the composite reward for each candidate query statement, including three dimensions of basic correctness, intermediate similarity and pattern alignment, and calculating the advantage value of the candidate query statement relative to the average reward in the group; updating the model output strategy according to the advantage value by optimizing the objective function of the grouping relative strategy optimization algorithm, and repeating the iteration until the model converges; when the model does not reach the performance indicator, return the candidate sample to the reinforcement training stage to continue optimizing the model parameters.

8. The method of claim 1, wherein, The verification of the predicted query statement in step 4) includes: calculating three indexes of execution accuracy, structure matching rate and field matching rate, and judging whether the predicted query statement is valid based on the index results; when any index does not reach the set threshold, the predicted query statement is returned to the reinforcement learning training stage of step 2) for model optimization.

9. The method of claim 1, wherein, The generation of the natural language description in step 5) includes: inputting the obtained structured query result into a language model to generate a natural language description based on the database query result. 10.A multi-modal based SQL query statement intelligent generation system, characterized in that, Comprise: A data set construction module is configured to obtain a natural language question, a database schema and a real query statement, parse the natural language question and the database schema, extract involved database tables, fields and values from the real query statement, generate a candidate visual entity based on the natural language question and the extracted content, obtain a candidate image related to the candidate visual entity from an external image source, perform three-layer correlation evaluation on the candidate image and select the highest-scored image as a target image, replace an explicit visual entity in the natural language question based on the target image, and form a training sample containing image directionality; A model training module is configured to train a multi-modal large model in stages based on the training sample generated by the data set construction module, including a supervised fine-tuning stage to enable the model to generate an output consistent with the real query statement based on the natural language question, the target image and the database schema, and a reinforcement learning stage to update the model generation strategy by multi-candidate SQL sampling, composite reward calculation and strategy optimization, and iterate until the model converges; An SQL generation and verification module is configured to receive a natural language question, a target image and a database schema, parse the natural language question and the database schema structure information, generate a predicted query statement, and perform execution verification, structure matching verification and field matching verification on the generated query statement, and determine an effective predicted query statement based on the verification results; A result feedback module is configured to feed back the verification results of the SQL generation and verification module to the model training module, and adjust the model generation strategy in the subsequent reinforcement training stage when the predicted query statement does not meet the verification indexes.

Images