A multi-level cooperation-based image dataset construction method, system and model training method

Abstract

The application provides a multi-level cooperative-based image dataset construction method, system and model training method. The image dataset construction method comprises: obtaining an initial image pair set; estimating the average depth of the pixel points corresponding to the foreground object in the source image and the target image and the consistency of the background respectively; based on the estimated average depth and the consistency of the background, screening the image pairs that meet the depth change condition and the background consistency condition from the initial image pair set to form a geometric-level image pair set; based on a large visual language model, screening the image pairs in which the foreground object exists and can be perceived from the geometric-level image pair set to form a semantic-level image pair set, and generating the natural language text corresponding to the screened image pairs; based on the average depth, determining the depth difference quantization label of the image pair, taking the natural language text as the depth difference text label, and thus obtaining the image dataset. The application can ensure large-scale production and improve sample quality.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system, and model training method for constructing image datasets based on multi-level collaboration. Background Technology

[0002] With the rapid development of artificial intelligence in computer vision and image generation, data-driven deep learning methods have gradually become mainstream. Existing image datasets are mainly constructed in the following ways: The first category is manual collection and annotation: This method requires acquiring image samples in a strictly controlled experimental environment, and then manually annotating them frame by frame. Although the data accuracy is high, it is time-consuming and costly, making it difficult to scale up. The second category is synthetic data based on 3D rendering: Researchers can use rendering engines such as Unity and Blender to generate images and their depth information, thereby flexibly controlling objects and scenes. However, synthetic data and real scenes have significant differences in texture, lighting, and semantic richness, making it difficult to avoid the difference between virtual and reality, resulting in a significant reduction in the model's performance in real-world environments. The third category is video frame extraction and direct pairing: This method constructs sample pairs by extracting consecutive frames from videos. It has a greater advantage in terms of scale. However, due to the lack of effective screening and annotation mechanisms, the data often contains situations where the foreground and background change simultaneously, object motion is insignificant, or semantics are irrelevant, resulting in a very high noise ratio, making it difficult to meet the requirements for high-quality training.

[0003] Therefore, how to improve sample quality while ensuring large-scale production, and how to automatically attach corresponding labels to datasets, has become an urgent problem to be solved in the field. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, system and model training method for constructing an image dataset based on multi-level collaboration, which can simultaneously solve the problems of limited scale and high data noise ratio that are common in the construction of existing image datasets, and automatically add labels to image pairs.

[0005] One aspect of the present invention provides a method for constructing an image dataset based on multi-level collaboration, the method comprising the following steps: Obtain an initial set of image pairs; wherein each image pair in the initial set of image pairs includes a source image and a target image, and the source image and the target image are image frames of the same foreground object at different positions in the same background from the same shooting perspective; Based on a pre-trained depth estimation model, the average depth of pixels corresponding to foreground objects in the source and target images is estimated respectively. Based on a background consistency estimation algorithm, the consistency of the background in the source and target images is evaluated. Based on the estimated average depth and background consistency results, image pairs that meet the depth variation and background consistency conditions are selected from the initial image pair set to form a geometric image pair set. Based on a large-scale visual language model, image pairs with perceptible depth differences in foreground objects are selected from a geometric-level image pair set to form a semantic-level image pair set, and natural language text corresponding to the selected image pairs is generated; wherein, the natural language text is used to indicate the foreground depth changes between the source image and the target image in each selected image pair. The depth difference quantization label of the image pair is determined based on the average depth, and the natural language text is used as the depth difference text label. Thus, an image dataset is constructed based on the semantic-level image pair set, the depth difference quantization label, and the depth difference text label.

[0006] In some embodiments of the present invention, after obtaining the initial set of image pairs, the method further includes: performing foreground segmentation processing on the source image and the target image respectively, and determining the foreground region and background region in the source image and the target image; Background consistency is characterized by background displacement and structural similarity between the source and target images.

[0007] In some embodiments of the present invention, the depth variation condition is that the average depth difference between the source image and the target image is greater than a preset depth threshold. The background consistency condition is that the background displacement is less than a preset displacement threshold and the structural similarity is greater than a preset similarity threshold.

[0008] In some embodiments of the present invention, based on a large visual language model, image pairs with perceptible depth differences in foreground objects are selected from a geometric-level image pair set to form a semantic-level image pair set, and natural language text corresponding to the selected image pairs is generated, including: Each image pair in the geometric-level image pair set is input into an edit detection-visual question answering system built on a large visual language model to filter out image pairs between the source and target images that can describe the editing operation. The edit detection-visual question answering system outputs the first confidence score for each image pair. The image pairs selected by the edit detection-visual question answering system are input into the depth perception-visual question answering system built on a large visual language model to filter out image pairs in which the foreground objects have perceptible depth differences, and the depth perception-visual question answering system outputs the second confidence score corresponding to each image pair; Image pairs selected by the deep perception-visual question answering system are used to form a semantic-level image pair set. The image pairs selected by the deep perception-visual question answering system are then input into an image description generation system built on a large-scale visual language model. The output is the natural language text corresponding to each image pair in the semantic-level image pair set.

[0009] In some embodiments of the present invention, the method further includes: When the edit detection-visual question answering system determines that the input image pair does not meet the first expected result and the first confidence score of the input image pair is lower than the first set confidence threshold, a feedback correction system based on a large visual language model is used to generate a first supplementary prompt to guide the edit detection-visual question answering system to re-select; and When the deep perception-visual question answering system determines that the input image pair does not meet the second expected result and the second confidence level of the input image pair is lower than the second set confidence level threshold, a second supplementary prompt is generated by a feedback correction system based on a large visual language model to guide the deep perception-visual question answering system to re-select.

[0010] In some embodiments of the present invention, determining the depth difference quantization label of an image pair based on the average depth includes: Based on the preset threshold range-depth change level mapping relationship, the corresponding depth change level is determined according to the threshold range to which the average depth difference between the source image and the target image belongs, and the determined depth change level is used as the depth difference quantification label.

[0011] In some embodiments of the present invention, the image dataset also includes foreground category labels, which are generated by a classifier to classify foreground objects in the source image and / or target image.

[0012] In some embodiments of the present invention, obtaining an initial set of image pairs includes: According to the preset frame interval rules, two different image frames are extracted from the original image sequence and used as the initial image pair to obtain the initial image pair set.

[0013] Another aspect of the present invention provides a method for training an image editing model, the method comprising the following steps: The training sample set is obtained based on the image dataset obtained by the method described in any of the above embodiments; The training sample set is input into the initial image editing model, and the model is fine-tuned through iterative updates to obtain the pre-trained image editing model.

[0014] Another aspect of the present invention provides a multi-level collaborative image dataset construction system, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method described in any of the above embodiments.

[0015] This invention proposes a multi-level collaborative method, system, and model training method for constructing image datasets. By introducing a geometric filtering stage and a semantic review stage based on a Large Vision-Language Model (LVLM), this multi-level collaborative mechanism leverages both depth estimation and background stability assessment with efficient geometric discrimination capabilities, and the semantic understanding and generation capabilities of LVLM, thereby achieving automated transformation from large-scale image pairs to high-quality image pair sets. Furthermore, this application can be combined with an automated annotation mechanism to automatically generate image datasets containing multi-dimensional labels based on image samples. This invention significantly improves data production efficiency and quality, providing a solid data foundation for subsequent 3D perception, controllable generation, and intelligent modeling tasks.

[0016] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0017] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1 This is a flowchart illustrating a method for constructing an image dataset based on multi-level collaboration in one embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the process of training an image editing model in one embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the module interaction of an image dataset construction system based on multi-level collaboration in one embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0022] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0023] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0024] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0025] In tasks involving 3D scene understanding and spatial modeling, existing solutions often suffer from limited image datasets, low quality, and missing annotations, which become the core bottleneck restricting model performance and generalization ability. Based on this, this application proposes a multi-level collaborative image dataset construction method, aiming to simultaneously solve the problems of limited image dataset size, high data noise ratio, and difficulty in automatic annotation in existing technologies. Specifically, it includes: in the preliminary screening stage, initial image pairs are collected from the original image sequence using a preset frame interval rule to provide a sufficient number of image pairs with potential editing relationships for subsequent steps; in the geometric filtering stage, automated screening at the geometric level is performed, including depth estimation and background stability assessment, so as to efficiently identify and screen geometric image pairs (also referred to as candidate image pairs in this application) that meet the condition of significant spatial changes in the foreground and static background; in the semantic review stage, based on a large-scale visual language model, the candidate image pairs are semantically judged to confirm the rationality and significance of the depth changes of the image pairs, and a natural language description that matches the depth changes of the foreground of the image pairs is generated; in the automatic labeling stage, labels indicating depth differences and object categories are added to the high-quality image pairs after screening, thereby forming an image dataset containing rich information.

[0026] The proposed geometric-semantic collaborative mechanism effectively addresses the issues of low quality and difficult screening in existing image dataset construction because images inherently possess differentiated characteristics at both spatial and semantic levels. Geometric changes in images (such as foreground displacement and depth differences) can often be quickly detected from pixel and depth distributions using low-complexity models, while semantic changes (such as object categories and action intentions) require cross-modal reasoning using models with visual language understanding capabilities. By combining the two, geometric filtering can quickly eliminate samples with unstable backgrounds or unrelated to motion at the physical level of the image, while semantic review further judges the semantic rationality and directional consistency of the spatial changes in the sample for the candidate image, thus forming a dual constraint of "geometric-semantic". This step-by-step filtering and verification from the underlying pixel patterns to the high-level semantic logic can fully utilize the structural consistency and semantic continuity features of the image itself, significantly reducing the noise ratio and ensuring the consistency of the generated data samples in terms of spatial variation, semantic coherence, and visual realism.

[0027] The proposed method for constructing an image dataset can be accomplished through an automated data filtering and processing pipeline (designed as a three-tiered pyramid structure including preliminary filtering, geometric filtering, and semantic review). By inputting one or more initial sets of image pairs, the pipeline ultimately outputs an image dataset containing tens of thousands of compliant samples. The pipeline executes as follows: Figure 1 The steps S110~S140 are shown.

[0028] Considering the changes in foreground depth in the image pair and subsequent model training, this application uses the terms "source image" and "target image" to distinguish different images in the image pair (similar terms such as "first image" and "second image" can also be used for distinction). During model training, one of the source image and the target image is used as the image before editing to input the initial image editing model to be trained, while the other is used as the image after editing to adjust the model parameters.

[0029] Step S110: In the preliminary screening stage, an initial image pair set is obtained. This initial image pair set includes at least one image pair, and each image pair in the initial image pair set includes a source image and a target image (i.e., an image pair can be formed from a source image and a target image). Each image pair in the initial image pair set can be represented as... , Represents the source image. This represents the target image.

[0030] The initial screening stage is the first step in the screening process. It is responsible for coarsely filtering the massive image frame data to quickly discover potential image pairs that meet the criteria of this stage, reducing the computational cost of subsequent processing. Since this application mainly obtains high-quality image datasets through screening, and the computation speed of this stage is relatively fast, large-scale screening can be performed to obtain a large number of initial image pairs. That is, the goal of the initial screening stage is to obtain a sufficient number of image pairs with potential editing relationships for subsequent fine filtering.

[0031] More specifically, the process of obtaining the initial image pair set can be as follows: Based on a preset frame interval rule, two different image frames are extracted from the original image sequence, and these two extracted image frames are used as the source image and the target image, respectively (i.e., the initial image pair is obtained), thus yielding the initial image pair set. In other words, an initial image frame sequence can be extracted from the original image sequence according to a preset frame interval rule, thereby forming the initial image pair set.

[0032] In the initial image pair, the source image and the target image can be image frames of the same foreground object located at different positions in the same background under the same shooting perspective. Therefore, in order to ensure that each image pair in the initial image pair set is an image pair with a changing foreground position and to reduce the workload, the original image sequence can also be a sequence of image frames of the same foreground object under the same shooting perspective and in the same background.

[0033] As an example, the original image sequence can be obtained by decoding one or more high-density, specific-category video sources. For instance, frame extraction tools such as FFmpeg can be used to decode the video sources into the original image sequence at a fixed frame rate (e.g., 10fps). Moreover, video source categories with obvious foreground object movement and relatively static backgrounds can be selected as specific categories, such as selecting video sources in categories like indoor speeches, desktop unboxing (e.g., high-definition product review videos), stop-motion animation, or documentaries for decoding to obtain the original image sequence.

[0034] In some embodiments of the present invention, the preset frame interval rule can be a limitation on the inter-frame sampling step size. The inter-frame sampling step size can be set adaptively to avoid excessively high inter-frame similarity (to avoid the foreground positions in the two extracted image frames being too similar). This application does not specifically limit it; for example, one frame can be taken every 5 image frames and added to the initial image frame sequence. Alternatively, the inter-frame sampling step size can be determined based on the intensity of motion of the foreground object in the original image sequence. The more intense the motion, the smaller the step size, so that the spatial variation between the source image and the target image in the image pair is within a reasonable threshold range, ensuring that the image pair entering the subsequent process has potential usability. Moreover, in addition to limiting the inter-frame sampling step size, the preset frame interval rule can also limit the video source category and motion scene (indoor or outdoor) corresponding to the original image sequence.

[0035] Furthermore, this application requires that the source image and target image extracted from the original image sequence be different image frames, and does not restrict the order of the source image and target image in terms of time frames (referring to the positions of the source image and target image in the original image sequence in the time dimension). That is, after extracting two image frames from the original image sequence, one can be used as the source image and the other as the target image.

[0036] Unless otherwise specified below, the source image and the target image described are the same image pair.

[0037] Steps S120 and S130 are the core of data filtering. Compared with the preliminary filtering stage, the geometric filtering stage and the semantic review stage have higher time costs and can perform refined filtering.

[0038] Building upon the initial screening stage, a geometric filtering stage can be used to further filter candidate image pairs that satisfy the condition of significant spatial variation in the foreground and relatively static background. This stage utilizes an efficient pre-trained depth estimation model and background consistency estimation algorithm to rigorously verify the geometric constraints of the initial image pairs, thereby achieving rapid filtering of foreground depth variation and background stability (i.e., background consistency) with low complexity, effectively ensuring physical plausibility. Step S120 performs geometric filtering, including step S121 for filtering image pairs that satisfy the depth variation condition and step S122 for filtering image pairs that satisfy the background consistency condition. Only when both conditions are met will the image pair undergo semantic review. If the image pair does not meet the requirements of stable background or foreground subject continuity, the initial image pair is discarded, ensuring that image pairs entering subsequent processes have potential usability.

[0039] In some embodiments of the present invention, after obtaining the initial set of image pairs, the image dataset construction method proposed in this application may further include: determining the foreground and background regions in the source image, and the foreground and background regions in the target image. For example, a foreground segmentation model can be used to perform foreground segmentation processing on the source image and the target image in the initial image pair respectively to generate a foreground mask that distinguishes the foreground and background regions (the background mask can be obtained by inverting the foreground mask), thereby determining the foreground and background regions in the source image and the target image. The foreground segmentation model can be a general segmentation model (Segment Anything Model, SAM), etc. SAM, with its powerful zero-shot segmentation capability, can accurately generate masks of foreground objects without additional training. The above-described method of using foreground segmentation to distinguish the foreground and background of an image is only an example; edge detection or deep learning models can also be used, and the present invention is not limited thereto.

[0040] In this context, each region in the image is defined by a specific set of pixels. Therefore, the foreground region can be the region / location where the pixels corresponding to the foreground object are located, and the background region can be the region / location where the pixels corresponding to the background are located. The foreground regions of the source image and the target image can be represented as follows: and The background regions corresponding to the source image and the target image can be represented as follows: and .

[0041] Step S121: Based on the pre-trained depth estimation model, estimate the average depth of the foreground region in the source image and the target image respectively, and based on the estimated average depth, select image pairs that meet the depth variation conditions from the initial image pair set.

[0042] More specifically, a depth estimation model can be used to estimate the average depth of the foreground region in the source and target images belonging to the same image pair, i.e., to estimate the average depth of the foreground region in the source image. and the average depth of the foreground region in the target image ,in, The depth is the average depth function. The depth estimation models used in this application include, but are not limited to, MiDaS. For example, Geowizard and PatchFusion can also be used. Moreover, this application does not specifically limit the method of estimating the average depth of the foreground region using the depth estimation model. For example, the depth map corresponding to the source image and the depth map corresponding to the target image can be determined first, and then the average depth of the foreground region can be determined according to the pixel position corresponding to the foreground object.

[0043] As an example, a preset depth threshold can be used. (For example, in normalized depth space) The image pairs that meet the depth variation criteria are filtered and retained. The depth variation criteria can be defined as the absolute value of the average depth difference between the source and target images being greater than a preset depth threshold. It can be expressed by the formula: ; in, , which represents the absolute value of the foreground depth difference between the source image and the target image.

[0044] This application does not impose specific limitations on the conditions for depth changes; it is sufficient that the depth difference between the source image and the target image can be reflected.

[0045] Step S122: Based on the background consistency estimation algorithm, evaluate the consistency of the background in the source image and the target image; and based on the estimated background consistency results, select image pairs that meet the background consistency conditions from the initial image pair set.

[0046] More specifically, background consistency can be characterized by the background displacement between the source and target images (referring to the displacement of corresponding background pixels in the source and target images) and the structural similarity index measure (SSIM) between background regions in the source and target images. (For the background region...) and The purpose of evaluating structural similarity is to ensure that the background content and structure remain unchanged, while the purpose of evaluating background displacement is to eliminate background displacement caused by minor camera movements or shaking. Therefore, background consistency estimation algorithms can include background displacement estimation algorithms and structural similarity algorithms.

[0047] For example, the formula for the structural similarity index can be expressed as: ; in, and These represent the average brightness of the background region in the source image and the target image, respectively. and These represent the contrast of the background regions in the source and target images, respectively. express and covariance, and All of them are relatively small constants.

[0048] This application does not specify the method for calculating background displacement. For example, an optical flow model (such as RAFT (Recurrent All-Pairs Field Transforms)) can be used to evaluate the background region in the source and target images, calculate the average optical flow, and use it as the estimation result of background displacement.

[0049] As an example, a preset background stability threshold (including displacement threshold) can be used. and similar threshold The image pairs that meet the background consistency condition are filtered and retained. The background consistency condition can be that the background displacement is less than a preset displacement threshold. And the structural similarity is greater than the preset similarity threshold. It can be expressed by the formula: ; in, This represents the average optical flow magnitude of the background region in the source and target images.

[0050] This application does not constitute a violation of the law. and The value can be specifically limited and can be adjusted adaptively according to needs, such as... , Pixels. Furthermore, this application does not specifically limit the background consistency condition; it is sufficient to filter out image pairs with consistent backgrounds.

[0051] only ,and Only when an image pair contains meaningful depth displacement and has a stable background is it considered to be preserved, forming a geometric image pair set. Therefore, each image pair in the geometric image pair set satisfies the condition of foreground depth change and background staticity.

[0052] Although the candidate image pairs selected in the geometric filtering stage meet the physical requirement of "moving foreground and static background", the changes in the foreground may be meaningless (such as random jitter). Therefore, a semantic review stage can be added to perform semantic discrimination on the candidate image pairs based on a large visual language model, confirm the rationality and significance of the foreground changes, and generate corresponding natural language descriptions for the image pairs that pass the review, so as to ensure the high semantic validity of data instances in the image dataset.

[0053] In the semantic review stage, step S130 can be performed to achieve semantic-level filtering: Step S130, based on a large-scale visual language model, selects image pairs with perceptible depth differences in the foreground objects from the geometric-level image pair set to form a semantic-level image pair set, and generates the natural language text corresponding to the selected image pairs. .

[0054] More specifically, the semantic-level filtering process is as follows: Each image pair in the geometric-level image pair set is input into an edit detection-visual question answering system built on a large-scale visual language model to filter out image pairs where there is a describable editing operation between the source and target images; the image pairs filtered by the edit detection-visual question answering system are input into a depth perception-visual question answering system built on a large-scale visual language model to filter out image pairs where there is a perceptible depth difference between the foreground objects; a semantic-level image pair set is formed based on the image pairs filtered by the depth perception-visual question answering system, and the image pairs filtered by the depth perception-visual question answering system are input into an image description generation system built on a large-scale visual language model to output the natural language text corresponding to each image pair in the semantic-level image pair set.

[0055] The Edit Detection-Visual Question Answering System, Depth Perception-Visual Question Answering System, and Image Description Generation System can all take image pairs and prompt text as input and output the question-and-answer results (which can be displayed in text or other forms). The prompt text for the Edit Detection-Visual Question Answering System instructs it to use computer vision to identify whether there is a purposeful, describable editing operation between the source and target images (belonging to the same image pair). (If the foreground only exhibits occlusion, distortion, or other anomalous events, it is a random spatial change, not a describable editing operation.) The prompt text for the Depth Perception-Visual Question Answering System, when editing operations exist, instructs it to use computer vision to determine whether there is a perceptible depth difference between the foreground objects in the source and target images (belonging to the same image pair) (i.e., whether the distance between the foreground object and the observer is closer or farther). The prompt texts for the Edit Detection-Visual Question Answering System and the Depth Perception-Visual Question Answering System can be customized; they only need to be able to filter out image pairs with perceptible depth differences in the foreground objects based on computer vision. In addition, the semantic-level image pair set formed by the image pairs selected based on the depth perception-visual question answering system refers to the use of computer vision to select image pairs with different foreground depths between the source image and the target image as image pairs in the semantic-level image pair set.

[0056] As an example, the prompt text corresponding to the image description generation system instructs the system to identify the foreground depth changes of each image pair in the semantic-level image pair set using computer vision (foreground depth change of an image pair refers to the difference in foreground object depth between the source and target images within the same image pair) and output the corresponding natural language text. For example, the natural language text output by the image description generation system can describe the foreground depth changes from the source image to the target image ("Move the potted plant forward", "Push the chair further away"). Furthermore, the image description generation system can be a standalone computer vision model or integrated with a depth-aware visual question-answering system. Additionally, if the accuracy requirements of the semantic review stage are not high, the semantic-level image pair set can be formed based on image pairs selected from the geometric-level image pair set by either the edit detection-visual question-answering system or the depth-aware visual question-answering system, without the other visual question-answering system performing its function.

[0057] In some embodiments of the present invention, since the screening processes of the edit detection-visual question answering system and the depth perception-visual question answering system are both semantic predictions, in order to improve robustness, this application also adopts a feedback correction mechanism in the semantic review stage: the feedback correction system based on a large visual language model collects and detects the outputs (including answers and confidence levels) of the edit detection-visual question answering system and the depth perception-visual question answering system. When the edit detection-visual question answering system and the depth perception-visual question answering system give a negative answer for the first time and the confidence level is not high, the feedback correction system generates a follow-up question prompt (e.g., "Please check the object in the center of the image again. Has its size and perspective changed?"), prompting the edit detection-visual question answering system and the depth perception-visual question answering system to re-examine, thereby effectively reducing the false negative rate.

[0058] More specifically, assuming the edit detection-visual question answering system can output the first confidence level corresponding to each image pair in the geometric image pair geometry, and the depth perception-visual question answering system can output the second confidence level corresponding to each image pair (the image pair selected by the edit detection-visual question answering system), the feedback correction mechanism can be as follows: when the edit detection-visual question answering system determines that the input image pair does not meet the first expected result and the first confidence level corresponding to the input image pair is lower than the first set confidence level threshold, a first supplementary prompt is generated using a feedback correction system based on a large visual language model to guide the edit detection-visual question answering system to re-select (so that the edit detection-visual question answering system detects the image pair again and outputs the corresponding secondary answer and confidence level); when the depth perception-visual question answering system determines that the input image pair does not meet the second expected result and the second confidence level corresponding to the input image pair is lower than the second set confidence level threshold, a second supplementary prompt is generated using a feedback correction system based on a large visual language model to guide the depth perception-visual question answering system to re-select (so that the edit detection-visual question answering system detects the image pair again and outputs the corresponding secondary answer and confidence level). The first expected result can be that there is no describable editing operation between the source and target images of the same image pair, and the second expected result can be that there is no perceptible depth difference between the foreground objects in the source and target images of the same image pair. This application does not specifically limit the first and second confidence thresholds, and they can be set arbitrarily. Furthermore, if the secondary responses output by the edit detection-visual question answering system and the depth perception-visual question answering system do not conform to the first or second expected result, the image pair is filtered out and not added to the semantic-level image pair set.

[0059] For example, image pairs can be input into the editing detection-visual question answering system and the depth perception-visual question answering system respectively, and direct questions can be asked. The system outputs "yes / no" results and corresponding confidence scores. When the feedback correction system detects that any result is "no" and the corresponding confidence score is lower than the threshold (e.g., 0.6), it can automatically generate supplementary prompts to guide the model to refocus on key areas or semantic differences in the image (e.g., "Please check if the central object is closer to the camera") and initiate a second round of questioning to obtain correction results, thereby reducing the false negative rate and improving the reliability of judgment.

[0060] As an example, step S130 requires scheduling multiple systems built on LVLM, which can be accomplished by setting a master control program. Furthermore, the edit detection-visual question answering system, the depth perception-visual question answering system, the image description generation system, and the feedback correction system can all be intelligent agent systems built on large-scale visual language models to perform the corresponding screening steps. In other words, step S130 is completed using a multi-agent system built on a large-scale visual language model.

[0061] For example, a multi-agent system built on a large visual language model may include a change detection agent (i.e., an edit detection-visual question answering system), a spatial reasoning agent (i.e., a depth perception-visual question answering system), a text generation agent, and a meta-reasoning and decision-making agent. The change detection agent determines whether there is a meaningful, non-random spatial change relationship between image pairs; the spatial reasoning agent, given that the spatial change relationship is confirmed, determines whether the change involves significant spatial or depth differences in the foreground object; and the text generation agent generates natural language text describing the depth changes of all approved image pairs. This ensures that the images are interpretable at the linguistic level for the samples; the meta-reasoning and decision-making agent integrates the preliminary judgment results of the change detection agent and the spatial reasoning agent and generates supplementary prompts according to preset logic to guide the agent to re-examine the image differences in order to reduce the false negative rate.

[0062] The semantic review stage can be based on a large-scale visual language model to further filter candidate image pairs at the semantic level and generate natural language descriptions with high semantic validity. , The qualitative description of the depth variation of foreground objects between the source and target images can be regarded as the depth difference text labels corresponding to the image pairs. Therefore, high-quality data instances can be obtained after step S130 is completed. By repeating this process, a deep editing dataset containing tens of thousands or even millions of samples can be built.

[0063] The automatic labeling stage executes step S140 to generate the deep editing dataset (natural language text). (As a dataset obtained as depth difference text labels) Additional labels are added: Step S140, determine the depth difference quantization labels for image pairs based on the average depth. Thus, an image dataset is constructed based on a set of semantic-level image pairs, depth difference quantization labels, and depth difference text labels.

[0064] More specifically, determining the depth difference quantization label for image pairs based on average depth includes: according to a preset threshold range-depth change level mapping relationship, based on the average depth difference between the source image and the target image (i.e., the foreground depth difference calculated in the geometric filtering stage). The threshold range to which the depth change belongs is used to determine the corresponding depth change level, which is then used as the depth difference quantization label. Both textual depth difference labels and quantized depth difference labels can be used to represent the depth difference between a foreground object and a target image. However, compared to the qualitative description of textual depth difference labels, quantized depth difference labels can be based on depth difference... Quantize the depth difference of foreground objects in the source and target images.

[0065] For example, the preset depth change threshold range-depth change level mapping relationship can be: if the difference Less than The depth change level is then "minor change"; if the difference is... Between and between( If the difference is..., then the depth change level is "moderate change"; if the difference is... Greater than If the depth change level is "drastic change", then the depth change level is "drastic change". The depth change threshold range-depth change level mapping relationship mentioned above is only an example and can be customized.

[0066] In some embodiments of the present invention, considering that the subsequent image dataset will be used to train the image editing model, in order to improve the generalization ability of the model, an image dataset containing different categories of foreground objects can be used for training. Therefore, the image dataset also includes the corresponding foreground category labels for each image pair. Foreground category labels are generated by a classifier that identifies foreground objects in the source and / or target images. For example, object categories such as "face," "vehicle," and "table and chair" can be used as foreground category labels.

[0067] As an example, the classifier can be a general classification model such as a classification agent built based on a large visual language model or a vector machine. If the classifier is a classification agent, the foreground object image obtained through foreground segmentation in step S120 can be input into the classification agent, along with corresponding prompt text (such as "Please identify the main object category in this image"). The output of the classification agent is then used as the foreground category label. The prompt text corresponding to the classification agent instructs the agent to identify the category of the foreground object in the input image.

[0068] This application does not specifically limit the order of steps for determining the foreground category label. For example, to save computing resources, the category of the foreground object in the filtered image pair can be determined after step S130, or the category of the foreground object in each image pair can be determined directly after step S110 is completed. This invention is not limited to this.

[0069] Through an automated marking mechanism, the image obtained in step S110 can be used to mark... The natural language text generated in step S130 And the depth change label obtained in step S140 (i.e., depth difference quantification label). and object category labels The system automatically labels each data instance (referring to each image pair in the set) in the semantic-level image pair set, forming a high-quality image dataset with rich metadata and multimodal features.

[0070] As an example, image datasets can be structured into quintuples. This is represented in the form of a , to support subsequent model training, validation and application.

[0071] The high-quality image dataset constructed by this invention can significantly improve the dataset's performance in terms of realism, structure, and diversity, providing strong data support for applications such as virtual reality, autonomous driving, naked-eye 3D, and generative artificial intelligence. Furthermore, in the model optimization stage, the high-quality, large-scale dataset generated in steps S110-S140 can be used to enhance the 3D spatial perception capabilities of the image editing model (especially the diffusion model), solving the problems of 3D perception and controllable generation in complex scenes through a data-driven approach. Therefore, this invention also proposes a training method for an image editing model, which includes the following steps: obtaining a training sample set based on the image dataset obtained by the method described in any of the above embodiments; inputting the training sample set into an initial image editing model, and fine-tuning the model through iterative updates to obtain a pre-trained image editing model. The training sample set includes at least images based on source images. Deep difference text tags and depth difference quantification labels The obtained data samples. The initial image editing model can be an existing latent diffusion model or a custom model built based on an existing diffusion model architecture; this invention is not limited to these.

[0072] As an example, to ensure that the initial image editing model to be trained simultaneously captures semantic, geometric, and positional information, the training sample set can be obtained by processing the source images in the image pair using an image encoder. Encode the image to obtain image feature vectors; use a position encoder to extract the position information of foreground objects in the source image. (Essentially, the center coordinates of the foreground mask and the camera pose) are encoded as position vectors. The image feature vectors and position vectors are concatenated and input into a multilayer perceptron, which maps them to M-dimensional features. These features are then extracted using a text encoder. as well as The features are used to form a token, and the token is concatenated with the M-dimensional features output by the multilayer perceptron to obtain the final embedding vector. Encoding all image pairs in the image dataset accordingly yields the embedding vector sequence. (N is the sequence length). Among them, the image encoder, position encoder and text encoder can all be built based on CLIP (Contrastive Language–Image Pre-training).

[0073] In some embodiments of the present invention, during the iterative update process, instead of employing the traditional, resource-intensive Full Fine-tuning method (i.e., without updating all weights of the model during fine-tuning), a more efficient and modern optimization strategy is implemented based on Low-Rank Adaptation (LoRA) fine-tuning technology. This allows for the injection of specific knowledge and capabilities of deep editing into the initial image editing model with extremely low parameter update costs. Specifically, the LoRA fine-tuning scheme can be as follows: for key modules in the initial image editing model, their parameters are frozen, a "LoRA adapter" is connected in parallel next to the module, and the adapter's parameters are updated. For example, for a diffusion model architecture based on a self-attention mechanism, the weight matrix of the cross-attention layer... (dimension is) This can be considered its key module; the parallel LoRA adapter can be composed of two low-rank matrices. (dimension) )and (dimension) It consists of ) and rank much smaller and (like =8 or 16). The forward propagation process of the initial image editing model can be represented as: ( This is the input feature vector of the cross-attention layer. (This is the output hidden feature vector obtained after weight matrix transformation), but the gradient is only backpropagated and the matrix is ​​updated. and The parameters. The advantage of this fine-tuning method is enormous: the number of parameters that need to be trained is reduced from hundreds of millions to millions, making it possible to fine-tune for several hours on a single consumer-grade GPU, and after training, the resulting LoRA weight file (containing only...) and The file size is very small (usually only a few MB to tens of MB), making it easy to distribute and manage.

[0074] As an example, the differential labels at different depths can also be quantized according to a preset training order. A corresponding subset of training samples is input into the initial image editing model to improve the stability of the model's performance. That is, by designing a training pipeline for fine-tuning the initial image editing model, the information contained in the training samples can be learned in stages.

[0075] For example, such as Figure 2 As shown, the training strategy can be specifically as follows: In the first training phase, basic stability training is performed, utilizing the training sample set... A subset of "minor changes" The initial image editing model undergoes preliminary fine-tuning (using LoRA fine-tuning technology) to teach it how to maintain background stability and high consistency in texture and lighting of the foreground object when the foreground object undergoes minor displacement. In the second training phase, significant adaptive training is performed, building upon the model obtained in the first phase (including the original weights and the LoRA weights from the first phase). Subsets of data with "moderate change" and "drastic change" Fine-tuning (also using LoRA fine-tuning technology) is then performed. Since the model already has basic stability, this stage allows the model to focus more on learning and simulating changes in perspective, size, and occlusion relationships caused by large depth changes (i.e., a wider range of depth editing) without easily destroying the overall structure of the image, thereby improving the robustness of the model and the stability of the generated results.

[0076] Furthermore, while the model obtained through LoRA fine-tuning boasts powerful performance, its parameters remain in high-precision FP32 or FP16 format, resulting in a large file size and high demands on GPU memory and computing units during inference. To address this issue, this invention also introduces post-training quantization technology to perform lightweight design on the trained model.

[0077] As an example, to make the LoRA-tuned model easier to deploy and apply, 8-bit quantization can be used to compress and accelerate the model, ultimately producing a lightweight and efficient image editing model that performs exceptionally well on deep editing tasks. The specific steps for lightweighting the self-attention-based diffusion model architecture using 8-bit quantization are as follows: Step S01, Sample Extraction: Divide the training sample set into a test set and a validation set. For example, randomly select a small portion of data (e.g., 100 samples) from the training sample set as the test set (the selected test samples can be based on...). The samples were uniformly selected for testing, and the remaining samples were used as the validation set.

[0078] Step S02, Calculate quantization parameters: Input the validation set into the fine-tuned model and record the numerical range (e.g., maximum absolute value) of the weight tensor and activation values ​​of each layer of the model. Based on this statistical information, calculate the optimal scaling factor for each layer to linearly map floating-point numbers to the 8-bit integer (INT8) range [-127, 127].

[0079] Step S03, Quantization Conversion: Apply the calculated scaling factor to convert all (or some key) weight matrices of the model from FP32 / FP16 to INT8 format.

[0080] The final model obtained after quantization has a storage volume of about 1 / 4 that of the original model, and on hardware that supports INT8 integer operations (such as NVIDIA's Tensor Cores), the inference speed can be improved by 1.5 to 3 times. Importantly, due to the use of a calibration-based static quantization scheme, the accuracy loss caused by this performance improvement is very small and almost imperceptible to the visual senses.

[0081] In some embodiments of the present invention, after LoRA fine-tuning and lightweight training are completed, the process of depth editing based on the image editing model can be as follows: the image to be changed and the target instruction are input into the trained image editing model, and the depth-changed image corresponding to the image to be changed can be output; wherein, the target instruction is used to instruct the image editing model on the degree of depth change of the foreground image in the image to be changed.

[0082] The image dataset construction method proposed in this application is applicable to depth adjustment tasks in image editing, such as naked-eye 3D and artistic creation, and has the following significant advantages: ① The preset frame interval rules can remove video segments that do not meet the conditions at the source, thereby reducing the computational overhead of subsequent processing.

[0083] ② Through the geometric filtering stage, the source and target images differ significantly only in the spatial relationships of foreground objects, while maintaining a high degree of consistency in the background region. Semantic review further confirms the samples and generates language at the semantic level, ensuring high semantic validity of the data. This "multi-level collaborative" design pattern not only ensures the efficiency of large-scale data production but also takes into account the high quality and low noise of the samples, filling the industry gap in the construction of high-quality image datasets. It can provide solid support for the application of artificial intelligence in scenarios such as 3D perception, virtual reality, and autonomous driving, and has significant scientific research and industrial value.

[0084] ③ Through the automatic labeling mechanism, the image dataset can reflect multi-dimensional information such as image depth changes and foreground object categories.

[0085] ④ The dataset construction method proposed in this application can provide a complete, efficient and practical image editing solution, from data source to final model deployment, comprehensively solving the key challenges faced in the field of image depth editing.

[0086] Corresponding to the above method, the present invention also provides an image dataset construction system based on multi-level collaboration. The system includes a computer device, which includes a processor and a memory. The memory stores computer programs / instructions, and the processor is used to execute the computer programs / instructions stored in the memory. When the computer programs / instructions are executed by the processor, the system implements the steps of the method described above.

[0087] like Figure 3 As shown, the multi-level collaborative image dataset construction system proposed in this application includes a data acquisition module, a geometric filtering module, a semantic review module, and an automatic labeling module. The data acquisition module executes step S110 to acquire an initial set of image pairs. The geometric filtering module executes step S120 to estimate the average depth of pixels corresponding to foreground objects in the source and target images, as well as the background consistency. Based on the estimated average depth and background consistency results, it selects image pairs from the initial set of image pairs that meet the depth variation and background consistency conditions. The geometric filtering module may integrate a foreground segmentation model, a depth estimation model, and a background consistency estimation algorithm (e.g., it may include an optical flow model). The semantic review module executes step S130 to obtain a semantic-level image pair set and the natural language text corresponding to each image pair in the semantic-level image pair set. The semantic review module is constructed based on a visual language model. The automatic labeling module executes step S140 to automatically generate and attach multi-dimensional labels such as depth difference and foreground object category to the image pairs that pass the semantic review, thereby constructing the image dataset.

[0088] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0089] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0090] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for constructing an image dataset based on multi-level collaboration, characterized in that, The method includes the following steps: Obtain an initial set of image pairs; wherein each image pair in the initial set of image pairs includes a source image and a target image, and the source image and the target image are image frames of the same foreground object at different positions in the same background under the same shooting perspective; Based on a pre-trained depth estimation model, the average depth of the pixels corresponding to the foreground objects in the source image and the target image is estimated respectively. Based on a background consistency estimation algorithm, the consistency of the background in the source image and the target image is evaluated. Based on the estimated average depth and background consistency results, image pairs that meet the depth variation condition and background consistency condition are selected from the initial image pair set to form a geometric image pair set. Based on a large-scale visual language model, image pairs with perceptible depth differences in foreground objects are selected from a geometric-level image pair set to form a semantic-level image pair set, and natural language text corresponding to the selected image pairs is generated; wherein, the natural language text is used to indicate the foreground depth changes between the source image and the target image in each selected image pair; The depth difference quantization label of the image pair is determined based on the average depth, and the natural language text is used as the depth difference text label. Thus, an image dataset is constructed based on the semantic-level image pair set, the depth difference quantization label, and the depth difference text label.

2. The method according to claim 1, characterized in that, After obtaining the initial set of image pairs, the method further includes: performing foreground segmentation processing on the source image and the target image respectively to determine the foreground region and background region in the source image and the target image; The consistency of the background is characterized by the background displacement and structural similarity between the source image and the target image.

3. The method according to claim 2, characterized in that, The depth change condition is that the average depth difference between the source image and the target image is greater than a preset depth threshold. The background consistency condition is that the background displacement is less than a preset displacement threshold and the structural similarity is greater than a preset similarity threshold.

4. The method according to claim 1, characterized in that, The process, based on a large-scale visual language model, involves selecting image pairs from a geometric-level image pair set that exhibit perceptible depth differences in foreground objects to form a semantic-level image pair set, and generating the corresponding natural language text for the selected image pairs. Each image pair in the geometric-level image pair set is input into an edit detection-visual question answering system built on a large-scale visual language model to filter out image pairs between the source image and the target image that can describe the editing operation, and the edit detection-visual question answering system outputs the first confidence score corresponding to each image pair; The image pairs filtered by the edit detection-visual question answering system are input into the depth perception-visual question answering system built on a large visual language model to filter out image pairs in which the foreground objects have perceptible depth differences, and the depth perception-visual question answering system outputs the second confidence score corresponding to each image pair; The image pairs selected by the deep perception-visual question answering system form a semantic-level image pair set. The image pairs selected by the deep perception-visual question answering system are then input into an image description generation system built on a large-scale visual language model, and the output is the natural language text corresponding to each image pair in the semantic-level image pair set.

5. The method according to claim 4, characterized in that, The method further includes: When the edit detection-visual question answering system determines that the input image pair does not meet the first expected result and the first confidence level of the input image pair is lower than the first set confidence threshold, a first supplementary prompt is generated using a feedback correction system based on a large visual language model to guide the edit detection-visual question answering system to re-select; and When the deep perception-visual question answering system determines that the input image pair does not meet the second expected result and the second confidence level of the input image pair is lower than the second set confidence level threshold, a second supplementary prompt is generated using a feedback correction system based on a large visual language model to guide the deep perception-visual question answering system to re-select.

6. The method according to claim 1, characterized in that, The process of determining the depth difference quantization label for the image pair based on the average depth includes: Based on the preset threshold range-depth change level mapping relationship, the corresponding depth change level is determined according to the threshold range to which the average depth difference between the source image and the target image belongs, and the determined depth change level is used as the depth difference quantification label.

7. The method according to claim 1, characterized in that, The image dataset also includes foreground category labels, which are generated by a classifier to identify foreground objects in the source and / or target images.

8. The method according to claim 1, characterized in that, The acquisition of the initial image pair set includes: According to the preset frame interval rules, two different image frames are extracted from the original image sequence and used as the initial image pair to obtain the initial image pair set.

9. A training method for an image editing model, characterized in that, The method includes the following steps: A training sample set obtained based on the image dataset obtained by the method of any one of claims 1-8; The training sample set is input into the initial image editing model, and the model is fine-tuned through iterative updates to obtain the pre-trained image editing model.

10. A system for constructing an image dataset based on multi-level collaboration, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 8.

Images