Model training data construction method and device, electronic equipment and computer readable storage medium

Abstract

The application provides a model training data construction method and device, electronic equipment and a computer readable storage medium. The method comprises: performing binocular data synthesis processing on a target monocular image to obtain a corresponding candidate binocular training sample, wherein the candidate binocular training sample at least comprises a synthetic left image, a synthetic right image, a target disparity and a confidence map; performing disparity evaluation on the synthetic left image and the synthetic right image to obtain a disparity evaluation result, and determining a first quality index according to the deviation between the disparity evaluation result and the target disparity; performing regional feature statistics on the confidence map to obtain a second quality index; and screening the candidate binocular training sample based on the first quality index and the second quality index to obtain a target binocular training sample, wherein the target binocular training sample is used for training of a binocular stereo matching model. Through the application, a binocular training data set with high geometric accuracy can be efficiently constructed.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and computer-readable storage medium for constructing model training data. Background Technology

[0002] Robotics and industrial vision systems typically require reliable depth information for environmental perception, localization, grasping, obstacle avoidance, and human-computer interaction. Traditional RGB-D cameras can acquire depth through methods such as Time-of-Flight (ToF) and structured light, but their active emission mechanisms are limited by factors such as working distance, ambient light, reflective materials, equipment cost, and interference from multiple cameras. In contrast, binocular vision recovers depth through the parallax between the left and right images, offering advantages such as passive imaging, a wider range of distances, and lower hardware costs, thus gradually becoming an important depth perception solution in robotics and industrial scenarios. However, high-performance binocular stereo matching models usually rely on large-scale, high-quality data with parallax ground truth for training. High-precision annotation of real binocular data is difficult to obtain, typically requiring LiDAR, RGB-D cameras, industrial calibration equipment, or complex acquisition systems to assist in generating ground truth depth values. These methods are costly and prone to errors at detail boundaries, weak texture areas, reflective areas, and distant areas, making it difficult to meet the high-precision parallax label requirements for large-scale model training. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for constructing model training data, which can efficiently construct a binocular training dataset with high geometric accuracy and close to the distribution of real business, thereby significantly improving the generalization ability and perception accuracy of the binocular stereo matching model in real-world scenarios.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a method for constructing model training data, including: Candidate monocular images are acquired, and the candidate monocular images are filtered to obtain the target monocular image; The target monocular image is subjected to binocular data synthesis processing to obtain corresponding candidate binocular training samples, wherein the candidate binocular training samples include at least a synthesized left image, a synthesized right image, target disparity, and a confidence map; Disparity evaluation is performed on the synthesized left image and the synthesized right image to obtain a disparity evaluation result. Based on the deviation between the disparity evaluation result and the target disparity, a first quality index is determined, wherein the first quality index is used to evaluate the geometric consistency between the synthesized left image and the synthesized right image. Regional feature statistics are performed on the confidence map to obtain a second quality index for evaluating the reliability of the target disparity; The candidate binocular training samples are screened based on the first quality index and the second quality index to obtain target binocular training samples, wherein the target binocular training samples are used for training the binocular stereo matching model.

[0005] This application provides an apparatus for constructing model training data, including: The acquisition module is used to acquire candidate monocular images; The filtering module is used to filter the candidate monocular images to obtain the target monocular image; The synthesis module is used to perform binocular data synthesis processing on the target monocular image to obtain corresponding candidate binocular training samples, wherein the candidate binocular training samples include at least a synthesized left image, a synthesized right image, target disparity, and a confidence map. The evaluation module is used to perform disparity evaluation on the synthesized left image and the synthesized right image to obtain the disparity evaluation result; The determining module is used to determine a first quality index based on the deviation between the disparity assessment result and the target disparity, wherein the first quality index is used to evaluate the geometric consistency between the synthesized left image and the synthesized right image; The statistics module is used to perform regional feature statistics on the confidence map to obtain a second quality index for evaluating the reliability of the target disparity; The filtering module is further configured to filter the candidate binocular training samples based on the first quality index and the second quality index to obtain target binocular training samples, wherein the target binocular training samples are used for training the binocular stereo matching model.

[0006] This application provides an electronic device, including: Memory is used to store executable instructions for a computer; The processor, when executing computer-executable instructions stored in the memory, implements the method for constructing model training data provided in the embodiments of this application.

[0007] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method for constructing model training data provided in this application.

[0008] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the method for constructing model training data provided in this application.

[0009] The embodiments of this application have the following beneficial effects: Through a closed-loop design involving pre-image quality purification, mid-stage binocular data synthesis, and post-stage multi-source geometric verification and sample grading, the geometric correctness of the synthesized image is verified using a first quality indicator without the need for expensive real disparity annotation. A second quality indicator is used to quantitatively assess the reliability of the target disparity itself. These two indicators complement each other and provide double checks, enabling the automatic and accurate filtering of geometrically inconsistent samples that are difficult to distinguish with the naked eye. This results in a high-quality binocular training set with high geometric accuracy and closely resembling real-world business distributions. This significantly reduces the training cost of the binocular stereo matching model and greatly improves the model's generalization and perception accuracy in complex real-world scenarios. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the architecture of the model training data construction system 100 provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application; Figure 3 This is a first flowchart illustrating the method for constructing model training data provided in this application embodiment; Figure 4 This is a schematic diagram of the second process of the method for constructing model training data provided in the embodiments of this application; Figure 5 This is a schematic diagram of the third process of the method for constructing model training data provided in the embodiments of this application; Figure 6 This is a schematic diagram illustrating the process of constructing model training data provided in the embodiments of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0013] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0014] In the following description, the terms “first, second, ...” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, ...” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0016] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0017] 1) Monocular image: refers to an image containing two-dimensional planar color and texture information of a scene, acquired using a single camera or a single-channel optical sensor. In 3D reconstruction technology, monocular images lack direct physical observations of depth or parallax, requiring deep learning or geometric prior models to deduce their 3D structure.

[0018] 2) Binocular stereo matching: This refers to the process of finding the corresponding pixels of the same physical 3D point in the two images (usually the left and right views) with a known relative positional relationship, and then calculating the horizontal projection position deviation (i.e., parallax) between the corresponding points in the two images, in order to recover the 3D geometric depth information of the scene.

[0019] 3) Parallax: refers to the horizontal pixel distance deviation between the projection points of the same physical 3D point on the left and right images of a binocular camera imaging system. Parallax is inversely proportional to the depth of the physical 3D point in the scene (i.e., the distance from the camera) and is a key geometric label for training binocular stereo matching models.

[0020] 4) Target disparity: refers to the disparity map that is output by the binocular data generation link in this embodiment of the application and is used as a reference ground truth or pseudo-label for the binocular stereo matching model to be trained. It is usually generated initially from monocular images through geometric or model inference and is used to guide the model to be trained to learn the geometric alignment relationship between the left and right images.

[0021] 5) Scale alignment: This refers to the process of converting and unifying the numerical distribution range of a disparity / depth map to the real physical scale space represented by another disparity / depth map by calculating the scaling factor and translation parameters, since different depth evaluation models may output geometric data with relative scale (without physical units) or different absolute scales.

[0022] 6) Forward mapping: This refers to an image projection technique that, based on known geometric disparity values, horizontally shifts each pixel in a reference image (e.g., the left view) in the projection direction (e.g., the right side) according to its corresponding disparity value, thereby reconstructing the initial target view (e.g., the right view) from the target's perspective. Due to occlusion and disparity discontinuities, the forward-mapped view often produces some missing pixels (i.e., holes).

[0023] In implementing the embodiments of this application, the applicant discovered that large-scale, high-quality synthetic data is crucial for the generalization ability of binocular stereo matching models. However, differences in appearance, texture, material, lighting, and camera distribution still exist between pure simulation data and real-world business scenarios. In related technologies, methods have been developed to generate synthetic binocular training data from monocular images. These methods typically first estimate geometric information using monocular depth or disparity models, then generate an initial right-view result through forward mapping, and finally use a diffuse image completion or fusion module to recover occluded areas. While these methods can reduce the cost of binocular data acquisition, the generated results may still have several problems: unstable input image quality can lead to monocular depth errors or completion artifacts; the synthesized right image may appear realistic in local areas but disrupt the geometric consistency between the left and right images; and the generated target disparity may also have errors in areas with weak texture, occlusion, reflection, and boundaries. Furthermore, generating binocular data from monocular images requires addressing issues of confidence, occlusion, and completion reliability. Therefore, simply generating synthetic stereo images is insufficient for training needs. A data filtering and quality assessment method tailored to specific business scenarios is also required. This method should select task-relevant, high-quality data sources suitable for synthesis from massive amounts of real or internet images, and automatically determine the suitability of samples for stereo model training using multi-source quality metrics after synthesis. This approach should simultaneously consider image quality, stereo matching consistency, pseudo-label confidence, and consistency between independent geometric models to construct a more reliable stereo training dataset.

[0024] In view of this, embodiments of this application provide a method, apparatus, electronic device, computer-readable storage medium, and computer program product for constructing model training data, which can efficiently construct a stereo training dataset with high geometric accuracy and close resemblance to real-world business distributions, thereby significantly improving the generalization ability and perception accuracy of the stereo matching model in real-world scenarios. The electronic device provided in the embodiments of this application will be described below. The electronic device provided in the embodiments of this application can be implemented as a server or as a terminal device.

[0025] In some embodiments, see Figure 1 , Figure 1 This is a schematic diagram of the architecture of the model training data construction system 100 provided in the embodiments of this application, as shown below. Figure 1As shown, the model training data construction system 100 provided in this application embodiment includes: server 200, network 300 and terminal device 400 (e.g., robot), wherein network 300 can be a local area network or a wide area network, or a combination of the two. The server 200 can acquire candidate monocular images and filter them to obtain target monocular images. Next, it can perform binocular data synthesis processing on the target monocular images to obtain corresponding candidate binocular training samples. These candidate binocular training samples include at least a synthesized left image, a synthesized right image, target disparity, and a confidence map. Subsequently, the server 200 can evaluate the disparity of the synthesized left and right images to obtain disparity evaluation results. Based on the deviation between the disparity evaluation results and the target disparity, a first quality index is determined, which can be used to evaluate the geometric consistency between the synthesized left and right images. Then, regional feature statistics can be performed on the confidence map to obtain a second quality index used to evaluate the reliability of the target disparity. Finally, the server 200 can filter the candidate binocular training samples based on the first and second quality indices to obtain target binocular training samples, which can be used to train a binocular stereo matching model. After obtaining the trained stereo matching model, the server 200 can send the trained stereo matching model to the terminal device 400 via the network 300, so that the trained stereo matching model can be deployed locally on the terminal device 400.

[0026] It should be noted that the method for constructing model training data provided in this application embodiment can also be implemented by the terminal device 400 alone. For example, the terminal device 400 can generate candidate binocular training samples based on its own computing power, and select target binocular training samples from the candidate binocular training samples. Then, the selected target binocular training samples are used to train the binocular stereo matching model to be trained.

[0027] The structure of the electronic device provided in the embodiments of this application will be further described below. Taking the electronic device as a terminal device as an example, see... Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application. Figure 2 The illustrated electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 540.

[0028] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0029] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0030] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0031] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0032] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0033] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. Presentation module 553 is used to enable the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with user interface 530. The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0034] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A device 555 for constructing model training data stored in memory 550 is shown. This device can be software in the form of programs and plug-ins, including the following software modules: acquisition module 5551, filtering module 5552, synthesis module 5553, evaluation module 5554, determination module 5555, statistics module 5556, prediction module 5557, alignment module 5558, and recognition module 5559. These modules are logically connected and can therefore be arbitrarily combined or further split according to the functions implemented. It should be noted that... Figure 2 For ease of explanation, all the above modules are shown at once, but this should not be regarded as an exclusion of the model training data construction device 555. It may only include the acquisition module 5551, the screening module 5552, the synthesis module 5553, the evaluation module 5554, the determination module 5555, and the statistics module 5556. The functions of each module will be explained below.

[0035] The method for constructing model training data provided in this application will be specifically described below with reference to exemplary applications and implementations of the electronic device provided in the embodiments of this application. It should be noted that the electronic device can be a server or a terminal device.

[0036] For example, see Figure 3 , Figure 3 This is a first flowchart illustrating the method for constructing model training data provided in this application embodiment, which will be combined with... Figure 3 The steps shown are explained.

[0037] In step 101, candidate monocular images are acquired.

[0038] In some embodiments, candidate monocular images can be obtained from business image libraries, industrial scene image libraries, internet image libraries, or open-source image datasets. The business image library may include an internal enterprise production scene image library, a warehouse scene image library, an industrial inspection scene image library, and a robot operation scene image library. The images in the internet image library may include publicly available internet image resources, image data from image and text datasets, and scene image data obtained by web crawling. The open-source image dataset may include publicly available natural image datasets, industrial scene image datasets, indoor scene image datasets, and robot vision datasets, etc.

[0039] In step 102, the candidate monocular images are filtered to obtain the target monocular image.

[0040] In some embodiments, step 102 can be implemented as follows: extracting image features of candidate monocular images using a pre-trained vision-text model, and extracting text features corresponding to the text description of the downstream task; determining the similarity between image features and text features, and selecting candidate monocular images with similarity greater than a similarity threshold as relevant monocular images related to the downstream task; obtaining image quality information corresponding to the relevant monocular images, and removing relevant monocular images that do not meet preset quality conditions based on the image quality information; and using the remaining relevant monocular images as target monocular images.

[0041] For example, after acquiring candidate monocular images, scene relevance screening can be performed on these images to retain those relevant to downstream tasks. For instance, a pre-trained visual-text model can be used to represent the candidate monocular images, and the matching degree between the images and the target scene can be determined by combining the text description of the downstream task. Specifically, the pre-trained visual-text model can be used to extract image features from the candidate monocular images; simultaneously, the text description corresponding to the downstream task can be obtained, and the visual-text model can be used to extract the text features corresponding to the text description; then, the similarity between the image features and the text features is calculated; candidate monocular images with a similarity greater than a preset similarity threshold are selected as relevant monocular images related to the downstream task. After completing scene relevance screening, image quality screening can be further performed on the relevant monocular images. For example, image quality information corresponding to relevant monocular images can be obtained, and relevant monocular images that do not meet preset quality conditions can be removed based on the image quality information (e.g., image sharpness greater than a sharpness threshold, watermark coverage ratio less than a coverage threshold, annotation coverage ratio less than a coverage threshold, border anomaly less than an anomaly threshold, image resolution greater than a resolution threshold, image aspect ratio within a preset aspect ratio range, overexposure less than an overexposure threshold, underexposure less than an underexposure threshold, contrast greater than a contrast threshold, and repetition less than a repetition threshold). The image quality information can include at least one of the following: blur level, watermark information, caption information, annotation coverage information, segmentation mask coverage information, detection box coverage information, black border information, white border information, abnormal border information, image resolution, image aspect ratio, overexposure, underexposure, image contrast, repetition, near-repetition, and image category attributes. After the above scene relevance filtering and image quality filtering, the remaining relevant monocular images can be used as target monocular images. Among them, the target monocular image has the characteristics of being relevant to the downstream task scene, having high image quality, and being suitable for geometric estimation and binocular data synthesis, thus providing a relatively reliable data foundation for the subsequent generation of candidate binocular training samples.

[0042] It should be noted that the pre-trained visual-text model can be a CLIP model, or any other model with image-text joint representation capabilities. The text description can be pre-defined based on the business scenario of the downstream task. For example, when the downstream task is related to industrial robots, warehousing and logistics, or factory automation, the text description can include one or more keywords, phrases, or sentence structures such as "factory," "robot," or "industry." Matching can be performed using only a single text description, or multiple text descriptions can be used to calculate similarity separately and then weighted, averaged, maximized, or jointly ranked. Furthermore, the similarity between image features and text features can be calculated using cosine similarity. Alternatively, dot product similarity, normalized relevance, or other similarity metrics that reflect the degree of image-text matching can be used. The similarity threshold can be set based on the quality of the data source, the coverage of the target scene, and the balance between recall and precision. Alternatively, instead of using a fixed threshold, a preset number or proportion of images ranked high based on similarity can be selected as relevant monocular images.

[0043] In step 103, the target monocular image is processed by binocular data synthesis to obtain the corresponding candidate binocular training samples.

[0044] Here, the candidate binocular training samples include at least the synthetic left image, the synthetic right image, the target disparity, and the confidence map.

[0045] In some embodiments, see Figure 4 , Figure 4 This is a schematic diagram of the second process of the method for constructing model training data provided in the embodiments of this application, as shown below. Figure 4 As shown, Figure 3 Step 103 shown can be achieved through Figure 4 Steps 1031 to 1035 shown are implemented, and will be combined with Figure 4 The steps shown are explained.

[0046] In step 1031, the relative disparity and absolute disparity of the target monocular image are obtained.

[0047] In some embodiments, a relative disparity estimation model can be used to obtain the relative disparity of a target monocular image. This model can be a deep neural network model that focuses on learning the relative geometry, perspective cues, and occlusion edges within the image. The input target monocular image is a color image with a preset height and width, and the output is a single-channel relative disparity map with the same spatial resolution as the input image. Since the relative disparity estimation model is trained using a normalized target or mixed dataset, the output relative disparity value does not represent the true physical camera disparity (i.e., pixel-level offset), but rather characterizes the relative foreground / background occlusion relationships and relative distance order between objects in the scene. Furthermore, to introduce a realistic physical scale, embodiments of this application can also input the same target monocular image into a pre-trained absolute depth estimation model to obtain absolute disparity with physical scale information. For example, in an industrial robot hand-eye camera scenario, this model can directly predict and output absolute distance values ​​in meters or millimeters.

[0048] In step 1032, the relative disparity and absolute disparity are scale-aligned to obtain the target disparity.

[0049] In some embodiments, since the relative disparity estimation model provides more precise predictions of image edges, weak textures, and detailed geometric structures, while the absolute disparity estimation model carries the correct physical scale but is prone to edge blurring, scale alignment is required to combine their advantages and obtain a high-precision target disparity with a reasonable physical scale. Scale alignment can be achieved by solving a univariate linear regression model to establish a linear mapping between the two models. Specifically, invalid prediction regions such as the sky and distant scenery in the image can be excluded first. Multiple pixels are then uniformly sampled within the corresponding valid regions of the two disparity maps to ensure the representativeness of the sample distribution. Next, these sampled points can be used as samples for a least-squares linear regression solution. For example, the reprojection error between the relative disparity at each sampled point, after scaling and adding an offset, and the corresponding absolute disparity can be calculated. By minimizing the sum of squares of the reprojection errors at all sampled points, the optimal scaling factor and the optimal offset can be obtained. After obtaining the optimal scaling factor and offset, the scaling factor can be multiplied by the entire relative disparity map and the offset can be added to complete the linear scaling of the entire map and obtain the final target disparity map.

[0050] In step 1033, the target monocular image is forward-mapped according to the target parallax to obtain the initial result of the right view, and the occluded or empty areas in the initial result of the right view are image-completed to obtain the completed right view.

[0051] In some embodiments, after obtaining a target disparity map of a reasonable scale, the target monocular image can be used as the synthesized left image (i.e., the left view). The initial result of the right view can be reconstructed using forward mapping technology. Specifically, each pixel in the target monocular image can be traversed first. For any pixel, the disparity value at its corresponding coordinate position in the target disparity map is read. Next, based on the parallel configuration principle of the binocular camera (i.e., the corresponding point in the right view should be horizontally to the left of the point in the left view), the target horizontal projection coordinates of the pixel in the right view are calculated. This is done by subtracting the corresponding disparity value from the pixel's horizontal x-coordinate in the left view, while keeping its vertical y-coordinate unchanged. Subsequently, the color value of the pixel can be copied and filled into the pixel position corresponding to the target horizontal projection coordinates in the right view. After traversing and projecting all pixels in the left view, the initial result of the right view can be obtained. Since the forward mapping process is a many-to-one or one-to-empty geometric projection, invalid areas due to geometric occlusion or lack of pixel mapping will inevitably occur in the initial result of the right view. To restore the missing geometric holes and occluded areas in the initial right view to a visually realistic and structurally sound complete image, embodiments of this application can also utilize a generative image inpainting algorithm to generate a complete right view. Specifically, the initial right view and the synchronously generated complete area mask can be input into a pre-trained image inpainting model. This allows the image inpainting model to use the regions marked as valid in the mask as contextual visual cues to learn prior information about texture, lighting, materials, and structure in the scene. Then, pixel-level predictions can be performed on the holes and occluded areas marked as needing completion in the mask, synthesizing and filling in visually coherent texture content, and outputting a complete right view with full color.

[0052] In step 1034, the initial result of the right view and the completed right view are fused together to obtain the composite right view.

[0053] In some embodiments, since the initial result of the right view obtained by forward mapping has the most realistic physical geometric projection relationship in the unoccluded area, and the completed right view has a coherent visual texture in the hole and occluded areas, the initial result of the right view and the completed right view can be fused to obtain the final synthetic right image.

[0054] In step 1035, the target monocular image is used as the synthesized left image, and the synthesized left image, the synthesized right image, the target disparity, and the confidence map output synchronously when generating the target disparity are combined into candidate binocular training samples.

[0055] In some embodiments, before performing step 1035, the following processes may also be performed: horizontally flipping the target monocular image and determining the horizontal flip consistency error based on the disparity evaluation results before and after flipping; visually evaluating the target monocular image using multiple different geometric evaluation models and determining the consistency deviation between the multiple visual evaluation results output by the multiple geometric evaluation models respectively; determining the scale alignment residual after scaling the relative disparity and absolute disparity; calculating the local gradient of the target disparity to obtain the local depth smoothing anomaly region; and determining the confidence map based on at least one of the horizontal flip consistency error, consistency deviation, scale alignment residual, and local depth smoothing anomaly region.

[0056] For example, a pixel-level confidence map can be obtained by performing multivariate geometric uncertainty and degradation feature analysis on the target monocular image. Specifically, the stability and symmetry of the geometric prediction can be tested by introducing horizontal image flipping. For instance, the target monocular image can first be horizontally mirrored to obtain a flipped monocular image. Then, a disparity evaluation model is called to re-evaluate the disparity of the flipped monocular image to obtain a flipped disparity map. To compare in the same physical space, the flipped disparity map can be horizontally mirrored again to obtain a reverse-aligned flipped disparity map. Subsequently, each pixel coordinate can be traversed one by one, and the absolute difference between the value of the original target disparity map at that coordinate and the value of the reverse-aligned flipped disparity map at the corresponding coordinate can be calculated. This absolute difference is used as the horizontal flipping consistency error of that pixel. Furthermore, to overcome the bias caused by the distribution defects of the training dataset or improper network architecture design of a single neural network, this application embodiment also introduces multiple geometric evaluation models with different architectures for joint verification. Specifically, the same target monocular image can be input into multiple preset geometric evaluation models in parallel, resulting in multiple initial disparity prediction maps. Then, the resolution and physical scale of the multiple initial disparity prediction maps can be normalized for the entire image. Subsequently, the statistical deviation of the multiple disparity prediction results at the same coordinate position can be calculated pixel by pixel to characterize the consistency deviation between the multiple visual evaluation results. In addition, after generating the target disparity map, the reprojection residual can be recalculated for each pixel position of the entire image. For example, the absolute disparity value transformed at the pixel coordinate can be subtracted from the transformation value after multiplying the relative disparity value at the same coordinate by the optimal scale scaling factor and superimposing the optimal offset. The absolute value of the difference between the two can be calculated, and this absolute value of the difference can be used as the scale alignment residual of the pixel. Alternatively, the first and second spatial derivatives of the target disparity map can be calculated. For example, the first-order differential operator can be used to calculate the first-order disparity gradient magnitude of each pixel in the horizontal and vertical directions to locate the real physical edges and depth breaks in the scene. Subsequently, the second-order differential operator or the local mean-variance operator can be used to perform local smoothness verification on the first-order gradient map. If, in a continuous region without occlusion and physical boundaries, the first-order gradient magnitude of a pixel exhibits high-frequency alternation and violent oscillation, or if the second derivative is significantly greater than a preset smoothness threshold, then the region can be determined to have artifact noise or abrupt changes, and these pixel sets can be marked as regions with local depth smoothness anomalies.After obtaining the error mappings of the above four dimensions, a multi-channel fusion mechanism can be used to construct the final pixel-level confidence map. For example, a negative exponential function or log normalization algorithm can be used to map the horizontal flip consistency error, multi-model consistency bias, scale alignment residual, and local depth smoothing anomaly index to a probability interval between 0 and 1. Then, the four normalized error probability distribution maps can be weighted and summed pixel by pixel, or their minimum values ​​can be taken directly at each coordinate position. Finally, the comprehensive confidence value corresponding to each pixel coordinate is calculated by fusion, and a pixel-level confidence map with the same resolution as the target monocular image is reconstructed and output.

[0057] In step 104, disparity assessment is performed on the synthesized left and right images to obtain the disparity assessment results.

[0058] In some embodiments, step 104 can be implemented by: performing disparity evaluation on the synthesized left image and the synthesized right image using a pre-trained stereo matching evaluation model to obtain a disparity evaluation result, wherein the stereo matching evaluation model differs from the stereo matching model to be trained in terms of model parameters, network structure, or model performance.

[0059] For example, to ensure that the disparity evaluation results output by the stereo matching evaluation model can serve as an objective and independent geometric scale benchmark, and thus effectively examine the geometric distortions that may be hidden in the generation process of the synthesized right image, the pre-trained stereo matching evaluation model and the stereo matching model to be trained in this embodiment can have significant differences in the following dimensions, thereby avoiding collinear prediction biases between the two. For example, there are differences in their network architecture and matching mechanisms, or differences in model parameter scale and computational complexity, or differences in the source of the training dataset. Next, the synthesized left image and the synthesized right image can be input in pairs into the pre-trained stereo matching evaluation model to perform high-precision pixel-level stereo matching inference. For example, the stereo matching evaluation model can first extract high-dimensional feature maps of the synthesized left image and the synthesized right image at multiple resolution scales through its feature extraction backbone network. Then, the extracted left and right feature maps can be used to calculate the correlation matrix between corresponding feature points in the horizontal projection direction, thereby constructing a three-dimensional four-dimensional geometric cost space. Subsequently, a high-dimensional 3D convolutional network can be used to aggregate global and local geometric contexts in the cost space, thereby filtering noise and smoothing matching discrepancies in textureless regions. Then, a soft maximum operator or layer-by-layer residual accumulation can be used to regress high-precision, pixel-level continuous disparity distribution values ​​within a preset maximum disparity search range. After inference by the stereo matching evaluation model, the disparity evaluation result can be output. This result can be a two-dimensional floating-point matrix, reflecting the physically matchable relationship between the synthesized left and right images at the level of human visual perception and neural network visual features.

[0060] In step 105, a first quality index is determined based on the deviation between the disparity assessment result and the target disparity.

[0061] Here, the first quality metric can be used to evaluate the geometric consistency between the composite left and composite right images.

[0062] In some embodiments, step 105 can be implemented by: determining the disparity deviation between the disparity evaluation result and the target disparity at the pixel level, and statistically analyzing the disparity deviation in at least one of the full image region, local region, boundary region, occluded region, and unoccluded region to obtain a first quality index, wherein the first quality index includes at least: an endpoint error index for measuring the overall pixel-level average disparity deviation; and an abnormal pixel ratio index for measuring the proportion of bad pixels whose disparity deviation exceeds a preset threshold.

[0063] For example, the disparity evaluation result and the target disparity map can be compared pixel by pixel in the same image coordinate system. For instance, each pixel coordinate in the synthesized left image can be traversed, and the evaluated disparity value of the disparity evaluation result at that pixel coordinate, as well as the target disparity value of the target disparity map at the corresponding pixel coordinate, can be read. Then, the absolute difference between these two disparity values ​​can be calculated, and this absolute difference can be used as the pixel-level disparity deviation at that pixel. After traversing all pixels, a two-dimensional single-channel disparity deviation map with the same spatial resolution as the synthesized left image can be constructed and output. In order to comprehensively verify the matching quality of the synthesized right image under different complex geometric scenes, this embodiment of the application can use geometric priors and semantic information to divide the disparity deviation map into multiple different physical regions (e.g., including the whole image region, local region, boundary region, occlusion region, etc.), and generate corresponding region selection masks. Subsequently, the constructed physical region masks can be overlaid onto the disparity deviation map, extracting only the disparity deviation values ​​within the corresponding mask-marked regions. The following two core quality metrics are then calculated: End-Point Error (EPE) and Bad Pixel Ratio / D1. In other words, the final output quality metric is not a single data point, but a structured set of multiple metrics, including, for example, the endpoint error of the entire image, the endpoint error of non-occluded areas, the bad pixel ratio of boundary regions, and the endpoint error of local regions of interest.

[0064] In step 106, regional feature statistics are performed on the confidence map to obtain a second quality index for evaluating the reliability of target disparity.

[0065] In some embodiments, the second quality indicator may include a first sub-quality indicator and a second sub-quality indicator, then see [link to relevant documentation]. Figure 5 , Figure 5 This is a schematic diagram of the third process of the method for constructing model training data provided in the embodiments of this application, as shown below. Figure 5 As shown, Figure 3 Step 106 shown can be achieved through Figure 5 The implementation of steps 1061 and 1062 shown will be combined with Figure 5 The steps shown are explained.

[0066] In step 1061, the central tendency of the confidence values ​​contained in the confidence plot is statistically analyzed within the entire plot range to obtain the first sub-quality index.

[0067] In some embodiments, each pixel coordinate corresponding to the synthesized left image (i.e., the target monocular image) in the confidence map can be traversed, the confidence value at each coordinate position can be extracted, and the first sub-quality index can be calculated using any of the following methods: calculating the global average confidence based on the arithmetic mean algorithm (for example, the confidence values ​​of all valid pixels in the entire image can be collected and the arithmetic mean of these values ​​can be calculated); calculating the median confidence based on the median algorithm (for example, the confidence values ​​of all valid pixels in the entire image can be sorted in ascending or descending order according to their size, and the value in the middle of the sorted sequence can be extracted as the median confidence).

[0068] In step 1062, the confidence map is binarized using a set confidence threshold, and the distribution characteristics of low-confidence pixels are statistically analyzed in the resulting binarized image to obtain the second sub-quality index.

[0069] Here, low-confidence pixels are pixels whose confidence values ​​are less than the confidence threshold.

[0070] In some embodiments, a hard thresholding technique can be used to first transform the continuously distributed pixel-level confidence map into a discrete binary state distribution map to isolate unreliable geometric regions. Specifically, a preset confidence threshold can be set first. Then, in the same image coordinate system, each pixel coordinate of the confidence map is traversed, and the confidence value at that coordinate is compared with the preset confidence threshold. If the confidence value at the current pixel coordinate is less than the preset confidence threshold, the point is determined to be a low-confidence pixel and marked with a value of 1 at the same coordinate position in the corresponding binary image; if the confidence value at the current pixel coordinate is greater than or equal to the preset confidence threshold, the point is determined to be a high-confidence pixel and marked with a value of 0 at the same coordinate position in the binary image. Next, at least one of the following spatial distribution statistics can be performed on the pixels marked with a value of 1 in the binarized image to generate the corresponding second sub-quality index: based on the maximum connected component area statistics (for example, a two-neighborhood connected component labeling algorithm can be run on the set of pixels with a value of 1 in the binarized image to identify and segment all low-confidence pixel blocks that are in contact with each other and adjacent to each other, calculate the number of pixels contained in each connected component, and filter out the connected component with the largest area. The proportion of the number of pixels in the connected component with the largest area to the total number of pixels in the entire image is determined as the second sub-quality index); based on spatial centroid and boundary distribution statistics (for example, the coordinate distribution of all pixels with a value of 1 in the binarized image can be statistically analyzed, and the geometric centroid coordinates of these low-confidence pixel sets can be calculated. Then, the proportion of pixels in these low-confidence pixel sets distributed in the lower half of the image and the region of interest in the center of the image can be statistically analyzed, and this proportion of pixels is used as the second sub-quality index).

[0071] In step 107, candidate binocular training samples are screened based on the first quality index and the second quality index to obtain target binocular training samples.

[0072] Here, the target binocular training samples can be used to train the binocular stereo matching model.

[0073] In some embodiments, before performing step 107, the following processing may also be performed: predicting the left image depth of the synthesized left image through a pre-trained monocular depth evaluation model, and after converting the left image depth into monocular disparity, scaling the monocular disparity with the disparity evaluation result to obtain a third quality index, wherein the third quality index is used to characterize the geometric consistency deviation between the monocular disparity and the disparity evaluation result. Step 107 can be implemented in the following way: when the first quality index, the second quality index and the third quality index all meet the preset joint judgment conditions, the corresponding candidate binocular training sample is used as the target binocular training sample.

[0074] For example, to overcome the potential system-level biases between the generation link and the stereo matching evaluation model, this application introduces a completely independent monocular depth evaluation model as a third-party referee. This monocular depth evaluation model can be pre-trained on an independent monocular depth dataset, and its network backbone structure or training objectives are different from any model in the generation link, ensuring that its prediction results have independent geometric priors. Next, the synthesized left image can be input into the monocular depth evaluation model to predict the corresponding left image depth map. After obtaining the left image depth map, the focal length parameters of the downstream target binocular camera and the physical baseline distance between the left and right cameras can be read. Using the inverse proportional imaging relationship in perspective projection geometry, the depth value of each pixel in the left image depth map is converted into the corresponding monocular disparity value, thereby reconstructing and generating a monocular disparity map. Because the scale output of the monocular depth assessment model may be offset or be a relative scale without physical units, while the disparity assessment result output by the stereo matching assessment model is strictly bound to the pixel scale, the two must be scale-aligned in the same scale space before calculating the bias. For example, the alignment parameter can be solved using least squares regression or the median ratio method. Then, the obtained alignment parameter is multiplied by each pixel value in the entire monocular disparity map to obtain the scale-aligned monocular disparity map. After scale alignment, the consistency between these two disparities from completely different technical approaches can be measured. For example, the absolute difference between the scale-aligned monocular disparity and the disparity assessment result can be calculated pixel by pixel in the same image coordinate system to generate a consistency error map. Then, the mean, median, or outlier pixel ratio of all values ​​in the consistency error map can be statistically analyzed across the entire image, and the statistically obtained value is determined as the third quality index. Subsequently, the first, second, and third quality indices calculated above can be used for multi-dimensional joint decision-making. For example, corresponding admission thresholds can be pre-set for each index, including: a first threshold for limiting binocular matching reconstruction error (for endpoint error or bad pixel rate in the first quality index); a second threshold for limiting the risk of spatial defect aggregation (for the proportion of the largest connected component after binarization in the second quality index); and a third threshold for limiting systematic geometric conflicts (for mono- and binocular consistency deviation in the third quality index). Then, the three indices of each candidate binocular training sample can be compared with their corresponding thresholds. The joint decision condition can employ strict cascaded constraints with logic gates, for example: the first quality index is less than the corresponding first threshold, the second quality index is less than the corresponding second threshold, and the third quality index is less than the corresponding third threshold.If the first, second, and third quality indicators all meet the aforementioned joint judgment conditions, the candidate binocular training sample can be determined as a high-fidelity sample with no geometric distortion and absolutely reliable pseudo-labels (i.e., target disparity). It can then be archived as a target binocular training sample in the training set database for accurate training of the subsequent binocular stereo matching model. If any one of the three indicators fails to meet the set conditions, a veto mechanism can be activated, directly determining the sample as unqualified. This sample can be removed, or its loss weight can be set to zero or reduced during training.

[0075] In other embodiments, after performing binocular data synthesis processing on the target monocular image to obtain corresponding candidate binocular training samples, the following processing can also be performed: identifying the synthesized left image to obtain the invalid region mask corresponding to the synthesized left image; combining the invalid region mask, performing at least one of the following processing on the candidate binocular training samples: excluding the region identified by the invalid region mask when determining the first quality index, the second quality index, or the third quality index; assigning a reduced training loss weight to the region identified by the invalid region mask when training the binocular stereo matching model using the obtained target binocular training samples; calculating the area ratio of the region identified by the invalid region mask in the entire image, and removing the corresponding candidate binocular training sample if the area ratio is greater than a preset area ratio threshold.

[0076] For example, because the real physical world contains many regions that do not conform to the geometric assumptions of stereo matching (such as the sky at infinite distances that cannot be measured, strong reflections that disrupt the continuity of local optical flow, etc.), these regions can introduce serious noise and instability into the matching learning of stereo matching models. Therefore, high-precision local region identification can be performed on the synthesized left image. For example, a pre-trained lightweight semantic segmentation model can be called to perform pixel-by-pixel semantic category inference on the synthesized left image, accurately segmenting the sky category region and the transparent physical category region. Then, the hole occlusion mask generated by the forward mapping in the previous steps can be directly extracted, and the pixel regions marked as generated by completion can be merged together. Then, the union operation can be performed on all the regions identified above that may have an unstable impact on stereo matching, thereby constructing a two-dimensional binarized invalid region mask with the same spatial resolution as the synthesized left image. When calculating the first quality metric (i.e., binocular matching error), the second quality metric (i.e., confidence metric), and the third quality metric (i.e., mono- and binocular consistency deviation), invalid region masks can first be used as spatial filters and superimposed onto the corresponding error maps (e.g., disparity deviation maps or consistency error maps). Then, all pixel coordinates can be traversed, and only the error values ​​at pixel locations in the invalid region mask corresponding to valid logical values ​​can be extracted and counted. For pixel coordinates with invalid logical values ​​in the mask, they are directly discarded, zeroed out, or excluded from the denominator when calculating the global arithmetic mean, median, or proportion of bad pixels. Furthermore, even if some candidate binocular training samples pass the joint decision screening, their internal local reflections and edge completion still have slight geometric uncertainties. To prevent the model from forcibly fitting these unstable local noises during training, leading to network parameter divergence or decreased generalization ability, local adaptive loss control can be implemented during training. For example, the loss weights of these samples can be reduced during training. In addition, in some extreme cases (such as when most areas of an image are invalid matching regions), even if a few valid regions are retained, they have no training value. Therefore, corresponding samples can be removed based on area ratio. In this way, large areas of degraded and meaningless redundant garbage data can be filtered out, which greatly saves the computational power consumption for subsequent model training and ensures the high purity and efficiency of the final training set.

[0077] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0078] The method for constructing model training data provided in this application can be applied to scenarios that require the construction of business-related binocular training data, such as robotics, industrial vision, autonomous driving, warehousing and logistics, and indoor navigation. The implementation can be an offline data filtering system, a data production pipeline, a training data storage system, a data cleaning module before model training, etc.

[0079] In some embodiments, see Figure 6 , Figure 6 This is a schematic diagram illustrating the process of constructing model training data according to an embodiment of this application, as shown below. Figure 6 As shown in the embodiments of this application, the method for constructing model training data mainly includes the following steps: data source acquisition and pre-screening, calling the stereo data generation module to generate candidate stereo training samples, stereo quality filtering and multi-source consistency evaluation, joint quality judgment and data storage. The above steps are described in detail below.

[0080] (a) Data source acquisition and pre-screening In some embodiments, candidate monocular images can be first obtained from industrial image libraries, industrial scene image libraries, internet image libraries, or open-source image datasets. Data sources can include real-world business images within an enterprise, publicly available image datasets, internet image and text datasets, images captured by robots, etc. To make the data more relevant to downstream tasks, scene relevance filtering can be performed based on business keywords, text descriptions, category labels, or image features. For example, a visual-text model can be used to extract image features and calculate the similarity between the image and downstream task keywords or text descriptions. For instance, for industrial robot scenarios, text descriptions such as factory, industry, workshop, robotic arm, warehouse, and metal parts can be used to filter images related to the business scenario. The similarity threshold can be dynamically adjusted based on the data source quality, recall rate, and filtering accuracy requirements. Subsequently, image quality pre-screening can be performed on the candidate images. This pre-screening includes, but is not limited to: blur detection; watermark, caption, bounding box, and segmentation box detection; black border, white border, and abnormal border detection; resolution detection; aspect ratio detection; overexposure, underexposure, and low contrast detection; deduplication of duplicate or near-duplicate images; filtering of unnatural images, stitched images, screenshots, UI images, and images with rendering anomalies; and semantic filtering, such as removing images with excessive sky coverage, excessive solid color background coverage, or no effective foreground target. After the above screening, target monocular images that are relevant to downstream business, have high image quality, and are suitable for geometric estimation and right view generation can be obtained.

[0081] (ii) Call the stereo data generation module to generate candidate stereo training samples In some embodiments, the filtered monocular image (i.e., the target monocular image) can be input into the binocular data generation module to obtain candidate binocular training samples. The binocular data generation module can employ any monocular-to-binocular synthesis method. For example, the binocular data generation module may include: obtaining the relative disparity of the input image using a relative disparity estimation model; obtaining the absolute disparity with scale information using an absolute depth estimation model; performing scale alignment on the relative disparity and the absolute disparity to obtain a target disparity with a reasonable scale; performing forward mapping based on the target disparity to generate an initial result for the right view; performing image completion on occluded or holed regions in the initial result for the right view; fusing the forward mapping result and the completion result to obtain a synthesized right image; and outputting data such as the synthesized left image, the synthesized right image, the target disparity, the confidence map, the non-occluded mask, and the completion region mask.

[0082] (III) Binocular quality filtration and consistency assessment In some embodiments, since the synthesized right image may have defects that are not obvious to the naked eye but can disrupt binocular matching training, this application embodiment can further perform multi-source quality assessment after generating candidate binocular training samples. This assessment does not only consider the visual quality of the image, but also simultaneously judges the consistency between target disparity, left and right image matching relationship, confidence distribution, and independent geometric estimation. Specifically, the synthesized left image and synthesized right image can be input into a pre-trained stereo matching assessment model to obtain disparity assessment results. The stereo matching assessment model can be FoundationStereo, IGEV, RAFT-Stereo, or other high-performance stereo matching models. Then, the disparity assessment results can be compared with the target disparity output by the binocular data generation module to obtain a binocular matching error index (i.e., the first quality index). This index includes, but is not limited to: EPE (used to measure overall pixel-level disparity deviation), D1 / Bad Pixel Ratio (used to measure the proportion of bad pixels with errors exceeding a threshold), local region error, boundary region error, and errors of occluded and unoccluded regions counted separately. For example, samples that simultaneously satisfy EPE < τ1 (i.e., the preset disparity deviation threshold) and D1 < τ2 (i.e., the preset bad pixel ratio threshold) can be retained. The significance of this step is that if the true matchability between the synthesized left and right images is inconsistent with the target disparity, even if the images look normal, the output of the stereo matching evaluation model will deviate significantly from the target disparity, thereby exposing hidden data quality problems.

[0083] Furthermore, when generating target disparity, the binocular data generation module simultaneously outputs a pixel-level confidence map. This confidence map can be used to characterize the reliability of target disparity in different regions. For example, the confidence map can be obtained through methods such as horizontal flip consistency, multi-model consistency, scale alignment residuals, occlusion relationships, or local depth smoothing anomalies. Statistical analysis can be performed on the confidence map to obtain sample-level quality indicators (i.e., secondary quality indicators), including but not limited to: overall image average confidence, median confidence, proportion of low-confidence regions, area of ​​the largest continuous low-confidence region, and foreground region confidence. For example, samples with an overall image average confidence greater than τ3 (i.e., a preset confidence threshold) and a low-confidence region proportion less than τ4 (i.e., a preset proportion threshold) can be retained. The significance of this step is that the target disparity generated from the link output typically comes from monocular geometric priors. Its advantages include structural integrity and good boundary details, but it may contain scale, occlusion, or texture-related errors in some areas. The stereo matching evaluation model, on the other hand, directly utilizes the correspondence between the synthesized left and right images for disparity estimation, reflecting whether the synthesized sample truly possesses learnable binocular geometric relationships. Therefore, the confidence map can be used to evaluate the reliability of the target disparity itself, while the stereo matching error can be used to evaluate the consistency between the binocular image and the target disparity; the two complement each other.

[0084] Furthermore, to further filter out systematic errors, embodiments of this application can introduce an independent monocular depth estimation model to re-predict the depth or disparity of the input left image. This independent monocular depth estimation model differs from the relative disparity model in the generation link, or uses different training data, different network structures, and different training objectives to reduce the systematic bias caused by the same model. Specifically, the independent monocular depth estimation model can be used to predict the depth of the left image; the depth of the left image can be converted into monocular disparity based on focal length, baseline, or normalization ratio; the disparity output by the stereo matching evaluation model can be scale-aligned with the monocular disparity; and the consistency error between the two can be calculated. If the consistency error is greater than a preset threshold, the sample is considered to have a target disparity error, a synthesized right image error, or a scene geometric anomaly, and thus is removed or its weight reduced. For example, the depth or disparity error after scale alignment can be directly compared, or the surface normal map can be calculated from the monocular depth and binocular estimation results respectively, and the normal consistency of the two can be compared. Normal consistency is less sensitive to absolute scale error and is more suitable for checking whether the local geometric structure is consistent.

[0085] It should be noted that for some scenarios, semantic segmentation, object detection, or rule-based methods can be used to generate invalid region masks. For example, the sky, highly reflective areas, transparent objects, solid-color walls, motion-blurred areas, occluded boundaries, and generated completion areas may have an unstable impact on binocular matching training. Therefore, the following processing can be performed on these regions: they should not be included in the sample-level quality score; quality indicators should be calculated separately; their weights should be reduced during training; and if the proportion of invalid regions is too high, samples should be removed.

[0086] (iv) Joint quality assessment and data entry In some embodiments, the above quality indicators can be jointly judged to obtain the final sample quality score or a determination of whether to retain it. The joint judgment rules may include: EPE < τ1, D1 < τ2, average confidence of the entire image > τ3, proportion of low-confidence regions < τ4, monocular / binocular disparity consistency error < τ5 (i.e., a preset consistency error threshold), normal consistency > τ6 (i.e., a preset normal consistency threshold), and proportion of invalid regions < τ7 (i.e., a preset area proportion threshold). Subsequently, based on the joint judgment results, the samples can be divided into: high-quality samples (directly enter the training set); medium-quality samples (can enter the training set, but with reduced weight during training); low-quality samples (removed); and abnormal samples (enter the error sample library for analyzing the failure modes of the generation module or filtering rules). The final data stored may include: composite left image; composite right image; target disparity; confidence map; non-occlusion mask; completed region mask; binocular evaluation disparity; quality scores such as EPE / D1; and metadata such as data source, scene category, and filtering version. This dataset can be used for pre-training, fine-tuning, distillation training, or quality analysis of stereo matching models.

[0087] In summary, the technical solutions provided by the embodiments of this application have the following beneficial effects: 1) The embodiments of this application can automatically filter images related to downstream scenarios from business image libraries, Internet image libraries or open source datasets, reducing the cost of manual collection and selection of business-related binocular training data.

[0088] 2) The embodiments of this application introduce image quality pre-screening before data generation, which can reduce the impact of low-quality inputs such as blur, watermark, annotation overlay, abnormal borders, and low resolution on monocular depth estimation and image completion, thereby reducing synthetic artifacts from the source.

[0089] 3) In this embodiment of the application, a stereo matching model is introduced for secondary evaluation after the data is generated. By comparing the evaluation disparity with the target disparity, indicators such as EPE and D1 can be used to discover binocular geometric errors that are difficult to identify with the naked eye but will affect training.

[0090] 4) The embodiments of this application combine the generated confidence map to judge the reliability of pseudo-labels, which can reduce the negative impact of target parallax in unstable areas such as occlusion, reflection, weak texture, and boundaries.

[0091] 5) The consistency check between the independent monocular depth model and the binocular evaluation results in this application embodiment can verify the reliability of the sample from another geometric source, reducing the risk of misjudgment caused by a single model or a single index.

[0092] 6) The embodiments of this application can combine sample-level screening and region-level weighting, which can not only remove bad samples, but also reduce the weight of some usable samples that have local problems, thereby improving data utilization.

[0093] 7) The dataset constructed in this application embodiment is closer to the real business distribution, which can alleviate the domain difference between pure simulation data and real industrial scenarios, and is conducive to improving the generalization ability of the binocular stereo matching model in real scenarios such as weak texture, reflection, industrial parts, and complex background.

[0094] The following description continues to illustrate the exemplary structure of the model training data construction apparatus 555 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules in the model training data construction device 555 stored in the memory 550 may include: an acquisition module 5551, a screening module 5552, a synthesis module 5553, an evaluation module 5554, a determination module 5555, and a statistics module 5556.

[0095] The module 5551 is used to acquire candidate monocular images; the module 5552 is used to filter the candidate monocular images to obtain target monocular images; the module 5553 is used to perform binocular data synthesis processing on the target monocular images to obtain corresponding candidate binocular training samples, wherein the candidate binocular training samples include at least a synthesized left image, a synthesized right image, target disparity, and a confidence map; the module 5554 is used to evaluate the disparity of the synthesized left image and the synthesized right image to obtain a disparity evaluation result; the module 5555 is used to determine a first quality index based on the deviation between the disparity evaluation result and the target disparity, wherein the first quality index is used to evaluate the geometric consistency between the synthesized left image and the synthesized right image; the module 5556 is used to perform regional feature statistics on the confidence map to obtain a second quality index used to evaluate the reliability of the target disparity; the module 5552 is also used to filter the candidate binocular training samples based on the first quality index and the second quality index to obtain target binocular training samples, wherein the target binocular training samples are used for training the binocular stereo matching model.

[0096] In the above scheme, the synthesis module 5553 is also used to obtain the relative disparity and absolute disparity of the target monocular image; to perform scale alignment on the relative disparity and absolute disparity to obtain the target disparity; to perform forward mapping on the target monocular image based on the target disparity to obtain the initial result of the right view, and to perform image completion on the occluded or hole regions in the initial result of the right view to obtain the completed right view; to perform fusion processing on the initial result of the right view and the completed right view to obtain the synthesized right image; to use the target monocular image as the synthesized left image, and to combine the synthesized left image, the synthesized right image, the target disparity, and the confidence map output synchronously when generating the target disparity into candidate binocular training samples.

[0097] In the above scheme, the synthesis module 5553 is also used to perform horizontal flipping processing on the target monocular image and determine the horizontal flipping consistency error based on the disparity evaluation results before and after flipping; to perform visual evaluation on the target monocular image through multiple different geometric evaluation models and determine the consistency deviation between the multiple visual evaluation results output by the multiple geometric evaluation models; to determine the scale alignment residual after scale alignment of relative disparity and absolute disparity; to perform local gradient calculation on the target disparity to obtain the local depth smoothing anomaly region; and to determine the confidence map based on at least one of the horizontal flipping consistency error, consistency deviation, scale alignment residual, and local depth smoothing anomaly region.

[0098] In the above scheme, the screening module 5552 is also used to extract image features of candidate monocular images through a pre-trained vision-text model, and extract text features corresponding to the text description of the downstream task; determine the similarity between image features and text features, and screen out candidate monocular images with similarity greater than the similarity threshold as relevant monocular images related to the downstream task; obtain image quality information corresponding to the relevant monocular images, and remove relevant monocular images that do not meet the preset quality conditions according to the image quality information; and use the remaining relevant monocular images as target monocular images.

[0099] In the above scheme, the evaluation module 5554 is further used to evaluate the disparity of the synthesized left image and the synthesized right image using a pre-trained stereo matching evaluation model to obtain a disparity evaluation result. The stereo matching evaluation model differs from the stereo matching model to be trained in terms of model parameters, network structure, or model performance. The determination module 5555 is further used to determine the disparity deviation between the disparity evaluation result and the target disparity at the pixel level, and to statistically analyze the disparity deviation in at least one of the full image region, local region, boundary region, occluded region, and unoccluded region to obtain a first quality index. The first quality index includes at least: an endpoint error index for measuring the overall pixel-level average disparity deviation; and an abnormal pixel ratio index for measuring the proportion of bad pixels whose disparity deviation exceeds a preset threshold.

[0100] In the above scheme, the second quality index includes the first sub-quality index and the second sub-quality index; the statistics module 5556 is also used to perform central tendency statistics on the confidence values ​​contained in the confidence map within the entire map range to obtain the first sub-quality index; the confidence map is binarized by a set confidence threshold, and the distribution characteristics of low confidence pixels are statistically analyzed in the obtained binarized image to obtain the second sub-quality index, wherein low confidence pixels are pixels with confidence values ​​less than the confidence threshold.

[0101] In the above scheme, the model training data construction device 555 further includes a prediction module 5557 and an alignment module 5558. The prediction module 5557 is used to predict the left image depth of the synthesized left image through a pre-trained monocular depth evaluation model. The alignment module 5558 is used to scale-align the monocular disparity with the disparity evaluation result after converting the left image depth into monocular disparity to obtain a third quality index, wherein the third quality index is used to characterize the geometric consistency deviation between the monocular disparity and the disparity evaluation result. The screening module 5552 is also used to select the corresponding candidate binocular training samples as target binocular training samples when the first quality index, the second quality index, and the third quality index all meet the preset joint judgment conditions.

[0102] In the above scheme, the model training data construction device 555 further includes an identification module 5559, which is used to identify the synthesized left image and obtain the invalid region mask corresponding to the synthesized left image; and, in combination with the invalid region mask, to perform at least one of the following processes on the candidate stereo training samples: when determining the first quality index, the second quality index, or the third quality index, exclude the region identified by the invalid region mask; when training the stereo matching model using the obtained target stereo training samples, assign a reduced training loss weight to the region identified by the invalid region mask; and calculate the area ratio of the region identified by the invalid region mask in the whole image, and if the area ratio is greater than a preset area ratio threshold, remove the corresponding candidate stereo training sample.

[0103] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, therefore it will not be repeated. For technical details not covered in the model training data construction apparatus provided in this application embodiment, please refer to... Figure 3 , Figure 4 ,or Figure 5 The meaning is understood in accordance with the description of any of the accompanying drawings.

[0104] This application provides a computer program product, which includes a computer program or computer-executable instructions. The processor of an electronic device reads the computer-executable instructions from a computer-readable storage medium, and executes the computer-executable instructions, causing the electronic device to perform the model training data construction method described above in this application.

[0105] This application provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they cause the processor to execute the method for constructing model training data provided in this application. For example, ... Figure 3 , Figure 4 ,or Figure 5 The method for constructing model training data is shown.

[0106] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0107] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0108] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be part of a file of other programs or data, such as in one or more scripts stored in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections).

[0109] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0110] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for constructing model training data, characterized in that, The method includes: Candidate monocular images are acquired, and the candidate monocular images are filtered to obtain the target monocular image; The target monocular image is subjected to binocular data synthesis processing to obtain corresponding candidate binocular training samples, wherein the candidate binocular training samples include at least a synthesized left image, a synthesized right image, target disparity, and a confidence map; Disparity evaluation is performed on the synthesized left image and the synthesized right image to obtain a disparity evaluation result. Based on the deviation between the disparity evaluation result and the target disparity, a first quality index is determined, wherein the first quality index is used to evaluate the geometric consistency between the synthesized left image and the synthesized right image. Regional feature statistics are performed on the confidence map to obtain a second quality index for evaluating the reliability of the target disparity; The candidate binocular training samples are screened based on the first quality index and the second quality index to obtain target binocular training samples, wherein the target binocular training samples are used for training the binocular stereo matching model.

2. The method according to claim 1, characterized in that, The step of performing binocular data synthesis processing on the target monocular image to obtain corresponding candidate binocular training samples includes: Obtain the relative disparity and absolute disparity of the target monocular image; The relative disparity and the absolute disparity are scale-aligned to obtain the target disparity; Based on the target parallax, the target monocular image is forward-mapped to obtain an initial result of the right view, and the occluded or empty areas in the initial result of the right view are image-completed to obtain a completed right view. The initial result of the right view and the completed right view are merged to obtain the composite right image; The target monocular image is used as the synthesized left image, and the synthesized left image, the synthesized right image, the target disparity, and the confidence map output synchronously when generating the target disparity are combined into the candidate binocular training samples.

3. The method according to claim 2, characterized in that, Before combining the synthesized left image, the synthesized right image, the target disparity, and the confidence map synchronously output when generating the target disparity into the candidate binocular training samples, the method further includes: The target monocular image is horizontally flipped, and the horizontal flip consistency error is determined based on the disparity evaluation results before and after the flip. The target monocular image is visually evaluated using multiple different geometric evaluation models, and the consistency deviation between the multiple visual evaluation results output by the multiple geometric evaluation models is determined. Determine the scale alignment residual after scaling the relative disparity and the absolute disparity; Local gradient calculation is performed on the target disparity to obtain local depth smoothing anomaly regions; The confidence map is determined based on at least one of the horizontal flip consistency error, the consistency deviation, the scale alignment residual, and the local depth smoothing anomaly region.

4. The method according to claim 1, characterized in that, The step of filtering the candidate monocular images to obtain the target monocular image includes: Image features of the candidate monocular images are extracted using a pre-trained vision-text model, and text features corresponding to the text descriptions of the downstream tasks are also extracted. Determine the similarity between the image features and the text features, and filter out candidate monocular images with a similarity greater than a similarity threshold as relevant monocular images related to the downstream task; Obtain the image quality information corresponding to the relevant monocular image, and remove the relevant monocular images that do not meet the preset quality conditions based on the image quality information; The remaining relevant monocular images are used as the target monocular image.

5. The method according to claim 1, characterized in that, The process of performing disparity evaluation on the synthesized left image and the synthesized right image to obtain a disparity evaluation result, and determining a first quality index based on the deviation between the disparity evaluation result and the target disparity, includes: The disparity of the synthesized left image and the synthesized right image is evaluated by a pre-trained stereo matching evaluation model to obtain the disparity evaluation result. The stereo matching evaluation model is different from the stereo matching model to be trained in terms of model parameters, network structure or model performance. The disparity evaluation result and the target disparity are determined at the pixel level, and the disparity deviation is statistically analyzed in at least one of the full image area, local area, boundary area, occluded area and unoccluded area to obtain the first quality index. The first quality index includes at least: an endpoint error index for measuring the overall pixel-level average disparity deviation; and an abnormal pixel ratio index for measuring the proportion of bad pixels whose disparity deviation exceeds a preset threshold.

6. The method according to claim 1, characterized in that, The second quality indicator includes a first sub-quality indicator and a second sub-quality indicator; The step of performing regional feature statistics on the confidence map to obtain a second quality index for evaluating the target disparity reliability includes: Within the entire graph, the central tendency statistics of the confidence values ​​contained in the confidence graph are performed to obtain the first sub-quality index. The confidence map is binarized by setting a confidence threshold, and the distribution characteristics of low-confidence pixels are statistically analyzed in the resulting binarized image to obtain the second sub-quality index. The low-confidence pixels are pixels with confidence values ​​less than the confidence threshold.

7. The method according to any one of claims 1 to 6, characterized in that, Before filtering the candidate binocular training samples based on the first quality index and the second quality index to obtain the target binocular training samples, the method further includes: The left image depth of the synthesized left image is predicted by a pre-trained monocular depth evaluation model. After converting the left image depth into monocular disparity, the monocular disparity is scale-aligned with the disparity evaluation result to obtain a third quality index. The third quality index is used to characterize the geometric consistency deviation between the monocular disparity and the disparity evaluation result. The step of filtering the candidate binocular training samples based on the first quality index and the second quality index to obtain the target binocular training samples includes: If the first quality indicator, the second quality indicator, and the third quality indicator all meet the preset joint judgment conditions, the corresponding candidate binocular training sample is used as the target binocular training sample.

8. The method according to any one of claims 1 to 6, characterized in that, After performing binocular data synthesis processing on the target monocular image to obtain corresponding candidate binocular training samples, the method further includes: The synthesized left image is identified to obtain the invalid region mask corresponding to the synthesized left image; Based on the invalid region mask, perform at least one of the following processes on the candidate stereo training samples: When determining the first quality indicator, the second quality indicator, or the third quality indicator, exclude the region identified by the invalid region mask. When training the stereo matching model using the obtained target stereo training samples, the region identified by the invalid region mask is assigned a reduced training loss weight. The area ratio of the region identified by the invalid region mask in the whole image is calculated. If the area ratio is greater than a preset area ratio threshold, the corresponding candidate binocular training sample is removed.

9. A device for constructing model training data, characterized in that, The device includes: The acquisition module is used to acquire candidate monocular images; The filtering module is used to filter the candidate monocular images to obtain the target monocular image; The synthesis module is used to perform binocular data synthesis processing on the target monocular image to obtain corresponding candidate binocular training samples, wherein the candidate binocular training samples include at least a synthesized left image, a synthesized right image, target disparity, and a confidence map. The evaluation module is used to perform disparity evaluation on the synthesized left image and the synthesized right image to obtain the disparity evaluation result; The determining module is used to determine a first quality index based on the deviation between the disparity assessment result and the target disparity, wherein the first quality index is used to evaluate the geometric consistency between the synthesized left image and the synthesized right image; The statistics module is used to perform regional feature statistics on the confidence map to obtain a second quality index for evaluating the reliability of the target disparity; The filtering module is further configured to filter the candidate binocular training samples based on the first quality index and the second quality index to obtain target binocular training samples, wherein the target binocular training samples are used for training the binocular stereo matching model.

10. An electronic device, characterized in that, include: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the method for constructing model training data according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, they implement the method for constructing model training data according to any one of claims 1 to 8.

Images