View-oriented implementation of multi-modal data augmentation system based on reverse thinking chain generation

Abstract

The application relates to a multi-modal data enhancement system based on reverse thinking chain generation for view analysis, comprising: an image context preprocessing module, which is used for introducing multi-view samples with the same attribute as context references for the quality problems existing in a single original image; a reverse visual reasoning generation module, which is used for guiding a multi-modal visual reasoning large model by adopting a task-oriented complete prompt word, and generating thinking chain data by combining a standard answer in an original data set; and a multi-dimensional verification post-processing module, which is used for constructing a data cleaning system based on two dimensions of rule verification and quality verification, and performing optimal screening on generated samples from the dimensions of thinking chain generation format, attribute quantity consistency and attribute name consistency, and multi-modal feature alignment degree and quality evaluation large model scoring dimensions. The application significantly reduces the data construction cost, and effectively improves the training effect of the visual reasoning large model for view analysis.

Description

Technical Field

[0001] This invention relates to the field of view parsing, and more particularly to image enhancement processing, specifically to a multimodal data enhancement system based on reverse thinking chain generation for view parsing. Background Technology

[0002] In the field of view parsing, the industry focuses on the task of describing the attributes of objects to generate high-quality datasets to support the training of large-scale visual reasoning models. These models, with their powerful multimodal feature alignment capabilities, demonstrate great potential in understanding complex object features. However, training a large-scale visual model capable of accurate logical reasoning requires not only a large number of image-label pairs but also thought chain data containing complete reasoning logic. Currently, existing technologies have significant limitations in constructing high-quality thought chain datasets for view parsing. Specifically, in practical applications and model training preparation, existing technologies mainly suffer from the following shortcomings: 1) Manual annotation of thought chain is extremely costly: Traditional image annotation only requires selecting the target or adding attribute labels, while thought chain data requires annotators to record the complete reasoning process "from visual features to attribute conclusions" in detail. This kind of annotation work with long text and strong logic is extremely time-consuming and labor-intensive. 2) Lack of suitable prompt words for open source models: Most existing open source multimodal large models are designed for training in general scenarios, and there are no ready-made prompt words to adapt to the professional needs of the view parsing field when generating thought chains. 3) Severe "illusion" in model output thought processes: During the automatic data generation process, large multimodal models are highly susceptible to "illusion" problems. Especially when the quality of surveillance images is poor, the model often fabricates details that do not exist in the image or generates logically contradictory reasoning paths. Summary of the Invention

[0003] This invention addresses key issues in the training of large-scale visual reasoning models in the field of view parsing, such as high annotation costs, lack of suitable cue words, and the tendency for model generation to produce illusions. Combining this with the attribute description task of the target in a surveillance scenario, it provides a multimodal data augmentation system based on reverse thinking chain generation for view parsing. This system solves the feature recognition problem caused by poor quality of the original surveillance images by introducing multi-view samples with the same attributes as contextual references. Furthermore, it innovatively employs a reverse reasoning mechanism based on standard answers, using task-oriented cue words to guide the large model to generate a thinking chain containing complete causal logic to obtain high-quality reasoning data. In addition, a dual-dimensional verification and cleaning system of rule validation and quality validation is constructed to rigorously screen and optimize the generated data, effectively suppressing the illusion problem of large-scale multimodal models and significantly improving the quality of the generated data.

[0004] To achieve the above objectives, the view-oriented parsing implementation of the present invention provides a multimodal data augmentation system based on reverse thought chain generation as follows: This view-oriented parsing implementation of a multimodal data augmentation system based on reverse thought chain generation is characterized by the following: The image context preprocessing module is used to address quality issues in a single original image by introducing multi-view samples with the same attributes as contextual references for feature enhancement. The reverse visual reasoning generation module, connected to the image context preprocessing module, is used to guide the multimodal visual reasoning model to perform reasoning using task-oriented complete prompts, and to perform reverse reasoning by combining the standard answers in the original dataset to generate thought chain data containing logical paths. The multi-dimensional verification post-processing module, connected to the reverse visual reasoning generation module, is used to construct a data cleaning system based on two dimensions: rule verification and quality verification. It selects the best samples from the dimensions of mind chain generation format, attribute quantity consistency, attribute name consistency, multimodal feature alignment, and quality evaluation model scoring.

[0005] Preferably, the image context preprocessing module specifically performs the following processing: Retrieve other viewpoint images with the same target attributes as the original image from the original dataset, and use these other viewpoint images as references for image context. Input them together with the original image into the multimodal visual reasoning model to achieve feature enhancement by supplementing the missing fine-grained appearance attributes and structural details in the original image.

[0006] Preferably, the reverse visual reasoning generation module specifically includes: The prompt word generation module is used to describe the corresponding reasoning task based on the image attributes, and to construct a special prompt word instruction that includes role settings, task objectives, reasoning templates and output specifications, thereby activating the visual perception and logical expression capabilities of the multimodal visual reasoning big model in the reconnaissance context. The thought chain generation module, connected to the prompt word generation module, is used to generate text information for the current target image based on the dedicated prompt word instructions. This text information contains a complete causal deduction process from the extraction of local visual features to the final target attribute answer. The text information is then input into the multimodal visual reasoning model to obtain thought chain data containing logical mapping relationships.

[0007] Preferably, the reverse visual reasoning generation module performs reverse inference specifically as follows: The target attributes labeled in the original image are input as prior constraints into the multimodal visual reasoning model, and the model is forced to backtrack and generate visual evidence and intermediate reasoning steps that lead to the conclusion, given the known conclusion.

[0008] Preferably, the multi-dimensional verification post-processing module performs rule verification processing in the following manner: Through a hard logic gating mechanism, the mind chain generation format verification, attribute quantity consistency verification, and attribute name consistency verification are performed sequentially. If any one of them fails, it is judged as a discarded sample. If each verification passes, the mind chain text content is retained.

[0009] Preferably, the thought chain generation format verification specifically includes: The generated text content is checked to see if it strictly follows the preset structured tags and to verify whether the text content order conforms to the preset logic. The preset logic is as follows: first, a complete reasoning step is generated for the generated text content, and then the number of target attributes inferred is summarized based on the text content.

[0010] Preferably, the attribute quantity consistency check specifically involves: extracting the count values ​​of various targets mentioned in the generated text content and comparing them with the target quantity in the original standard answer; The attribute name consistency check specifically involves matching the attribute names in the generated text content with the predefined attribute list in the original dataset to verify whether the attribute descriptions conform to the predefined specifications.

[0011] Preferably, the multi-dimensional verification post-processing module performs quality verification processing in the following manner: For the verified thought chain text content, an evaluation strategy combining multimodal feature alignment calculation and quality evaluation large model scoring is adopted. For multiple thought chain candidate samples generated from the same original image, the one with the highest score is selected as the final valid data through comparative analysis, thus achieving the best of the best.

[0012] Preferably, the multimodal feature alignment calculation specifically includes the following processing: A pre-trained image-text matching model is used to extract the visual feature vector of the original image and the text feature vector of the generated thought chain, respectively. The cosine similarity between the two is calculated to measure whether the generated thought chain description correctly restores the original image.

[0013] Preferably, the scoring of the large-scale quality assessment model specifically includes the following processing: A discriminative large model is constructed specifically for evaluating the quality of image resolution. Multiple candidate samples of thought chains generated for the same single original image are optimized. By comparing and analyzing the differences between different candidate samples in terms of logical coherence, detail richness, and factual accuracy, the one with the highest score is selected as the final valid data.

[0014] The view-oriented parsing implementation of the present invention enables a multimodal data augmentation system based on reverse thought chain generation, which can effectively solve the problem of lack of high-quality thought chain data in the training of existing visual reasoning models. Through image context enhancement, construction of task-oriented prompts, and multi-dimensional verification post-processing, this system can automatically generate logically rigorous structured descriptive data, significantly reducing data construction costs and effectively improving the training effect of large-scale visual reasoning models based on view parsing. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the framework structure of the view-oriented parsing implementation of the multimodal data augmentation system based on reverse thinking chain generation according to the present invention.

[0016] Figure 2 This is a schematic diagram of the image context preprocessing module of the present invention.

[0017] Figure 3 This is a schematic diagram of the structure of the reverse visual reasoning generation module of the present invention.

[0018] Figure 4 This is a schematic diagram of the structure of the multidimensional verification post-processing module of the present invention.

[0019] Figure 5 This is a schematic diagram illustrating the generation of thought chain data in a specific embodiment of the present invention. Detailed Implementation

[0020] To more clearly describe the technical content of the present invention, the following description is provided in conjunction with specific embodiments.

[0021] Before describing the embodiments of the present invention in detail, it should be noted that, in the following, the terms “comprising,” “including,” or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0022] like Figure 1 As shown, the view-oriented parsing implementation of this invention, based on reverse thinking chain generation, is a multimodal data augmentation system mainly comprising three functional modules: image context preprocessing, reverse visual reasoning generation, and multidimensional verification postprocessing. First, the present invention uses an image context preprocessing module (i.e., module 1, such as...) Figure 2 (As shown) Feature enhancement addresses quality issues present in single original images. Since surveillance images in view resolution are often affected by shooting distance, lighting, or environmental factors, resulting in targets being too small, features being blurry, or key information being occluded. This module retrieves other images with the same target attributes from the original dataset (i.e., multi-view samples) and uses them as contextual references, inputting them into the model along with the original image. By supplementing the missing fine-grained appearance attributes and structural details in the original image, it helps the large-scale multimodal visual reasoning model establish correct visual perception, thereby solving the reasoning error problem caused by unclear image features.

[0023] Furthermore, the quality problems existing in a single original image refer to situations where the target size is too small, the features are blurred, or key information is obscured due to factors such as shooting distance, lighting, or environmental factors.

[0024] Furthermore, the aforementioned multi-view samples with the same attribute refer to other viewpoint images in the original dataset that have the same target attribute as the original image. Specifically, an image that is the same as the true conclusion of the current target attribute is randomly selected from the original dataset as a positive sample reference image, and an image that is opposite to the true conclusion of the current target attribute is selected as a negative sample reference image, thus forming a comparative context reference together with the original image.

[0025] Furthermore, the aforementioned context reference refers to inputting the original image and multiple view samples with the same attributes into the model to supplement the missing semantic feature information in the original image.

[0026] As a preferred embodiment of the present invention, the semantic feature information used in this technical solution refers to the fine-grained appearance attributes and structural details of the target of interest in the view parsing scene. This information enables the multimodal visual reasoning large model to establish correct visual perception of the original image.

[0027] As a preferred embodiment of the present invention, the present invention generates a reverse visual reasoning module (i.e., module 2, such as...). Figure 3 (As shown) Generates thought chain data containing logical paths. The core of this module lies in "reverse reasoning," which uses the target attribute answer in the original labeled data as a priori constraint, forcing the model to backtrack and generate visual evidence and intermediate reasoning steps leading to the known conclusion. Specific implementations include: 1) Construct task-oriented prompts through the prompt generation module (i.e., module 2-1): Design dedicated instructions for attribute description tasks, including role settings, task objectives, inference templates, and output specifications, to activate the model's logical expression capabilities. Then, use the original image, the retrieved positive sample image, the retrieved negative sample reference image, and the attribute truth values ​​already labeled in the image as prior conditions to fill in the constructed task-oriented dedicated prompts.

[0028] 2) Generate thought chain data through the thought chain generation module (i.e., module 2-2): Guide the model to explicitly describe the logical mapping relationship between the region of interest in the image and its corresponding target attribute description, generating a complete causal deduction process from local visual features to the final target attribute answer. Specifically: This generates text containing a causal derivation process from local visual feature extraction to the final target attribute answer, explicitly describing the logical mapping relationship between the region of interest in the image and its corresponding target attribute description. Let the original image be labeled I, and the positive sample image be denoted as I0. + The negative sample reference image is denoted as Let y denote the attribute truth value, P denote the specific prompt word, and T denote the model. Then, the thought chain data r generated for this attribute can be represented as: r=T(P(I, I) + , ,y)) In a specific embodiment of the present invention, an example of the generated thought chain data r is as follows: Figure 5 As shown.

[0029] As a preferred embodiment of the present invention, the present invention utilizes a multi-dimensional verification post-processing module (module 3, such as...). Figure 4 (As shown) Construct a data cleaning system based on rule validation and quality validation to automatically screen generated samples: 1) Rule validation module (i.e., module 3-1): Performs the following checks in sequence: mind chain generation format validation, attribute quantity consistency validation, and attribute name consistency validation. If any one of these checks is not met, the sample is deemed discarded.

[0030] 2) Quality Verification Module (Module 3-2): After the rule verification is passed, a strategy combining multimodal feature alignment calculation and quality evaluation model scoring is further adopted. Specifically, for multiple thought chain candidate samples generated from the same original image, the highest-scoring one is selected as the final valid data through comparative analysis, achieving the best of the best. Finally, to eliminate "pseudo-reasoning" in the model-generated text that appears reasonable but lacks visual anchoring, the image-text matching model re-evaluates the highest-scoring thought chain to determine whether it follows the visual features of the original image.

[0031] As a preferred embodiment of the present invention, the aforementioned thought chain generation format verification refers to detecting whether the generated text strictly follows the preset structured tags, verifying whether its content order conforms to the first generation of complete reasoning steps, and then summarizing the number of order target attributes deduced.

[0032] In a preferred embodiment of the present invention, the attribute quantity consistency verification refers to extracting the count values ​​of various targets mentioned in the generated text and comparing them with the target quantity in the original standard answer, wherein the original standard answer is the attribute value annotated manually.

[0033] In a preferred embodiment of the present invention, the attribute name consistency verification refers to matching the attribute names in the generated text with the predefined attribute list in the original dataset to verify whether the attribute descriptions conform to the predefined specifications.

[0034] As a preferred embodiment of the present invention, the present invention filters the generated samples based on the quality verification dimension, which means that after the rule verification is passed, a soft evaluation strategy combining multimodal feature alignment calculation and quality evaluation large model scoring is further adopted.

[0035] The multimodal feature alignment refers to the calculation of the cosine similarity between the visual feature vector of the original image and the text feature vector of the generated thought chain, which are extracted by a pre-trained image-text matching model. This cosine similarity is used to measure whether the generated thought chain description faithfully reproduces the image content.

[0036] The aforementioned quality evaluation large model scoring refers to constructing a discriminative large model specifically for evaluation, selecting the best candidate sample of multiple thought chains generated for the same single original image, and choosing the one with the highest score as the final valid data by comparing and analyzing the differences between different candidate samples in terms of logical coherence, detail richness and factual accuracy.

[0037] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0038] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution device.

[0039] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0040] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0041] In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "embodiment," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0042] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

[0043] In summary, the view-parse-oriented multimodal data augmentation system based on reverse thought chain generation proposed in this invention effectively solves the problem of insufficient high-quality thought chain data in the training of existing visual reasoning models. Through image context enhancement, construction of task-oriented prompts, and multi-dimensional verification post-processing, this system can automatically generate logically rigorous structured descriptive data, significantly reducing data construction costs and effectively improving the training effect of large-scale visual reasoning models based on view parsing.

[0044] In this specification, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and variations can be made without departing from the spirit and scope of the invention. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. A view-oriented parsing implementation of a multimodal data augmentation system based on reverse thought chain generation, characterized in that, The system includes: The image context preprocessing module is used to address quality issues in a single original image by introducing multi-view samples with the same attributes as contextual references for feature enhancement. The reverse visual reasoning generation module, connected to the image context preprocessing module, is used to guide the multimodal visual reasoning model to perform reasoning using task-oriented complete prompts, and to perform reverse reasoning by combining the standard answers in the original dataset to generate thought chain data containing logical paths. The multi-dimensional verification post-processing module, connected to the reverse visual reasoning generation module, is used to construct a data cleaning system based on two dimensions: rule verification and quality verification. It selects the best samples from the dimensions of mind chain generation format, attribute quantity consistency, attribute name consistency, multimodal feature alignment, and quality evaluation model scoring.

2. The view-oriented parsing implementation of the multimodal data augmentation system based on reverse thought chain generation according to claim 1, characterized in that, The image context preprocessing module specifically performs the following processing: Retrieve other viewpoint images with the same target attributes as the original image from the original dataset, and use these other viewpoint images as references for image context. Input them together with the original image into the multimodal visual reasoning model to achieve feature enhancement by supplementing the missing fine-grained appearance attributes and structural details in the original image.

3. The view-oriented parsing implementation of the multimodal data augmentation system based on reverse thought chain generation according to claim 1, characterized in that, The reverse visual reasoning generation module specifically includes: The prompt word generation module is used to describe the corresponding reasoning task based on the image attributes, and to construct a special prompt word instruction that includes role settings, task objectives, reasoning templates and output specifications, thereby activating the visual perception and logical expression capabilities of the multimodal visual reasoning big model in the reconnaissance context. The thought chain generation module, connected to the prompt word generation module, is used to generate text information for the current target image based on the dedicated prompt word instructions. This text information contains a complete causal deduction process from the extraction of local visual features to the final target attribute answer. The text information is then input into the multimodal visual reasoning model to obtain thought chain data containing logical mapping relationships.

4. The view-oriented parsing implementation of the multimodal data augmentation system based on reverse thought chain generation according to claim 3, characterized in that, The reverse visual reasoning generation module performs reverse inference in the following specific ways: The target attributes labeled in the original image are input as prior constraints into the multimodal visual reasoning model, and the model is forced to backtrack and generate visual evidence and intermediate reasoning steps that lead to the conclusion, given the known conclusion.

5. The view-oriented parsing implementation of the multimodal data augmentation system based on reverse thought chain generation according to claim 3, characterized in that, The multidimensional verification post-processing module performs rule verification processing in the following manner: Through a hard logic gating mechanism, the mind chain generation format verification, attribute quantity consistency verification, and attribute name consistency verification are performed sequentially. If any one of them fails, it is judged as a discarded sample. If each verification passes, the mind chain text content is retained.

6. The view-oriented parsing implementation of the multimodal data augmentation system based on reverse thought chain generation according to claim 5, characterized in that, The specific format verification for the mind chain generation is as follows: The generated text content is checked to see if it strictly follows the preset structured tags and to verify whether the text content order conforms to the preset logic. The preset logic is as follows: first, a complete reasoning step is generated for the generated text content, and then the number of target attributes inferred is summarized based on the text content.

7. The view-oriented parsing implementation of the multimodal data augmentation system based on reverse thought chain generation according to claim 5, characterized in that, The aforementioned attribute quantity consistency check specifically involves: extracting the count values ​​of various targets mentioned in the generated text content and comparing them with the target quantity in the original standard answer; The attribute name consistency check specifically involves matching the attribute names in the generated text content with the predefined attribute list in the original dataset to verify whether the attribute descriptions conform to the predefined specifications.

8. The view-oriented parsing implementation of the multimodal data augmentation system based on reverse thought chain generation according to claim 5, characterized in that, The multi-dimensional verification post-processing module performs quality verification in the following manner: For the verified thought chain text content, an evaluation strategy combining multimodal feature alignment calculation and quality evaluation large model scoring is adopted. For multiple thought chain candidate samples generated from the same original image, the one with the highest score is selected as the final valid data through comparative analysis, thus achieving the best of the best.

9. The view-oriented parsing implementation of the multimodal data augmentation system based on reverse thought chain generation according to claim 8, characterized in that, The multimodal feature alignment calculation specifically includes the following processes: A pre-trained image-text matching model is used to extract the visual feature vector of the original image and the text feature vector of the generated thought chain, respectively. The cosine similarity between the two is calculated to measure whether the generated thought chain description correctly restores the original image.

10. The view-oriented parsing implementation of the multimodal data augmentation system based on reverse thought chain generation according to claim 8, characterized in that, The scoring of the large-scale quality evaluation model specifically includes the following processes: A discriminative large model is constructed specifically for evaluating the quality of image resolution. Multiple candidate samples of thought chains generated for the same single original image are optimized. By comparing and analyzing the differences between different candidate samples in terms of logical coherence, detail richness, and factual accuracy, the one with the highest score is selected as the final valid data.

Images