Binocular model training method, binocular stereo matching method, equipment and medium

Abstract

The invention discloses a binocular model training method, a binocular stereo matching method, equipment and a medium, and the method comprises the steps: obtaining a binocular image data set which comprises a binocular image sample, the binocular image samples comprise a real binocular image sample, a binocular image sample generated by a real monocular image and a binocular image sample generated by a simulation renderer in real time; and training the initial binocular stereoscopic vision model based on the binocular image sample to obtain a target binocular stereoscopic vision model. According to the technical scheme of the invention, in order to alleviate the problem of lack of binocular training data sets, the real binocular image sample can be obtained, the binocular image sample is generated by the real monocular image, and the binocular image sample is generated by the simulation renderer in real time, so that the expansion of the training data set is realized, and then a high-performance target binocular stereoscopic vision model is trained.

Description

Technical Field

[0001] The present invention relates to the field of binocular stereo vision technology, and in particular to a binocular model training method, a binocular stereo matching method, equipment and a medium. Background Art

[0002] Binocular stereo vision is a method of reconstructing the geometric structure of a 3D scene by calculating a disparity map from two RGB images. This technology is widely used in robot vision, augmented reality (AR), autonomous driving, and 3D modeling.

[0003] Traditional binocular stereo matching methods are usually based on a pipeline of cost calculation, cost aggregation, disparity optimization and post-processing. These methods face many challenges when dealing with complex scenes, such as:

[0004] 1. Reflections and lighting changes: Reflective surfaces may cause matching failures, while lighting changes will interfere with cost calculations.

[0005] 2. Perspective deformation: The difference in camera perspective will cause the same object to appear different in shape and size in the two images, thus affecting the matching.

[0006] 3. Weak texture areas: In areas with scarce texture information or repeated textures, it is difficult to find reliable matching points.

[0007] 4. Transparent objects: Due to the optical properties of transparent materials, parallax estimation may fail completely.

[0008] With the rise of deep learning, binocular stereo vision networks based on deep learning have made significant progress, such as end-to-end disparity estimation methods using models such as convolutional neural networks (CNNs) and visual transformers. Such methods effectively solve many problems in traditional methods, such as handling weak texture areas and complex lighting through feature extraction and global context modeling. However, the widespread application of deep learning methods still has some bottlenecks that need to be solved, especially at the data level:

[0009] 1. Training data is difficult to obtain: Binocular stereo vision models usually require large-scale high-quality training datasets that contain accurate disparity or depth annotations. However, generating such datasets in real scenes is extremely costly, especially when annotating transparent or dynamic objects.

[0010] 2. Poor domain adaptability: After the model is trained in a specific scenario, it is often difficult to adapt to new application scenarios, resulting in the need for additional data enhancement or transfer learning techniques. Summary of the invention

[0011] The present invention provides a binocular model training method, a binocular stereo matching method, a device and a medium, so as to improve the generalization ability of the model, enhance the robustness of the model, improve the training efficiency and data utilization rate, and promote the wide application.

[0012] According to one aspect of the present invention, a method for training a binocular model is provided, comprising:

[0013] Acquire a binocular image data set, wherein the binocular image data set includes binocular image samples, and the binocular image samples include real binocular image samples, binocular image samples generated from real monocular images, and binocular image samples generated in real time by a simulation renderer;

[0014] An initial binocular stereo vision model is trained based on the binocular image samples to obtain a target binocular stereo vision model.

[0015] In a possible implementation, obtaining a binocular image sample generated from a real monocular image in a binocular image dataset includes:

[0016] Acquire the real monocular image;

[0017] Determine a disparity map to be used based on the real monocular image and a preset reference;

[0018] According to the disparity map to be used, the real monocular image is transformed to obtain an image to be processed corresponding to the real monocular image, wherein the image to be processed is an image with a missing area;

[0019] Extracting features from the real monocular image to obtain real monocular image features, and filling missing areas in the image to be processed based on the real monocular image features to obtain a target image;

[0020] The real monocular image and the target image are determined as binocular image samples generated by the real monocular image.

[0021] In a possible implementation, the binocular image samples include a sample left image and a sample right image, and the training of an initial binocular stereo vision model based on the binocular image samples to obtain a target binocular stereo vision model includes:

[0022] Inputting the sample left image and the sample right image into the initial binocular stereo vision model to obtain a predicted disparity map output by the initial binocular stereo vision model;

[0023] Calculating a loss value based on the predicted disparity map and the annotated disparity map corresponding to the binocular image sample;

[0024] The model parameters of the initial binocular stereo vision model are updated based on the loss value to obtain the target binocular stereo vision model.

[0025] In a possible implementation, during the process of training the initial binocular stereo vision model, the method further includes:

[0026] Counting the prediction accuracy of the initial binocular stereo vision model for the binocular image samples corresponding to various scene types;

[0027] The rendering configuration parameters of the simulation renderer are adjusted in real time based on the prediction accuracy, so as to adjust the number of binocular image samples corresponding to various scene types generated by the simulation renderer.

[0028] In a possible implementation, after obtaining the target binocular stereo vision model, the method further includes:

[0029] Acquire a target binocular image sample in a target application scenario; wherein the target binocular image sample includes a target sample left image and a target sample right image;

[0030] Inputting the target sample left image and the target sample right image into the target binocular stereo vision model to obtain a predicted left disparity map and a predicted right disparity map output by the binocular stereo vision model;

[0031] Performing a consistency check based on the predicted left disparity map and the predicted right disparity map, and determining a fine-tuning binocular image sample from the target binocular image sample based on a result of the check;

[0032] The target binocular stereo vision model is fine-tuned based on the fine-tuning binocular image samples.

[0033] In a possible implementation manner, the performing consistency check based on the predicted left disparity map and the predicted right disparity map includes:

[0034] Performing coordinate transformation on the target sample left image based on the predicted left disparity image to obtain a target transformed right image;

[0035] Performing coordinate transformation on the target transformation right image based on the predicted right disparity image to obtain a target transformation left image;

[0036] The pixel error between the target transformed left image and the target sample left image is calculated. If the pixel error is less than a preset pixel error threshold, it is determined that the verification has passed; otherwise, it is determined that the verification has failed.

[0037] In a possible implementation manner, determining a fine-tuning binocular image sample from the target binocular image sample based on a verification result includes:

[0038] If the verification is passed, the left image of the target sample and the right image of the target sample when the verification is passed are used as the fine-tuning binocular image samples.

[0039] According to another aspect of the present invention, a binocular stereo matching method is provided, the binocular stereo matching method comprising:

[0040] Acquire a binocular image to be processed corresponding to a target scene, and input the binocular image to be processed into a target binocular stereo vision model obtained by using a binocular model training method;

[0041] Obtaining a target disparity prediction result output by the target binocular stereo vision model;

[0042] A binocular stereo matching result of the binocular image to be processed is determined based on the target disparity prediction result.

[0043] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0044] at least one processor;

[0045] and a memory communicatively connected to the at least one processor; wherein,

[0046] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the binocular model training method or binocular stereo matching method described in any embodiment of the present invention.

[0047] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the binocular model training method or binocular stereo matching method described in any embodiment of the present invention when executed.

[0048] The technical solution of the embodiment of the present invention aims to solve the problems of insufficient data and limited generalization in the process of training binocular stereo vision models. By combining the advantages of real binocular data and large-scale simulation data, a multi-source data fusion strategy and an automatic fine-tuning strategy based on "shadow mode" are proposed, which significantly improves the versatility and generalization ability of the model in a variety of scenarios, thereby constructing a high-performance binocular stereo vision model suitable for complex real environments.

[0049] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 A flowchart of a binocular model training method provided in Embodiment 1 of the present invention;

[0052] Figure 2 A flow chart of generating a binocular image sample from a real monocular image provided in the first embodiment of the present invention;

[0053] Figure 3 A flowchart of adjusting rendering configuration parameters of a simulation renderer in real time provided in Embodiment 1 of the present invention;

[0054] Figure 4 A flowchart of a binocular model training method provided in Embodiment 2 of the present invention;

[0055] Figure 5 A flowchart for fine-tuning a target binocular stereo vision model provided in the second embodiment of the present invention;

[0056] Figure 6 A consistency check flow chart provided for the second embodiment of the present invention;

[0057] Figure 7 A flowchart of a binocular stereo matching method provided in Embodiment 3 of the present invention;

[0058] Figure 8 A schematic diagram of the structure of a binocular model training device provided in Embodiment 4 of the present invention;

[0059] Fig. 9 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0061] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0062] Embodiment 1

[0063] Figure 1 This is a flowchart of a binocular model training method provided in the first embodiment of the present invention. This embodiment aims to solve the problems of insufficient data and generalization limitation in the process of training binocular stereo vision models. The method can be executed by a binocular model training device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method specifically comprises the following steps:

[0064] S110: Obtain a binocular image dataset.

[0065] Among them, the binocular image dataset is used to train the binocular stereo vision model. The binocular image dataset includes binocular image samples. The binocular image samples include real binocular image samples, binocular image samples generated from real monocular images, and binocular image samples generated in real time by a simulation renderer.

[0066] Understandably, the training of binocular stereo vision models usually relies on high-quality binocular image datasets with parallax annotations. However, the cost of collecting such data in real scenes is extremely high, especially when dealing with scenes such as dynamic objects, complex lighting, or transparent materials, the difficulty of data collection and annotation is further increased. Therefore, the existing real binocular image datasets are limited in scale, and the scenes and content are limited, resulting in insufficient generalization ability of the trained models and poor performance in diverse real-world scenes.

[0067] In order to build a high-performance binocular stereo vision model, the embodiment of the present invention adopts a multi-source data fusion strategy, combining real binocular image samples, binocular image samples generated from real monocular images, and binocular image samples generated in real time by a simulation renderer, so as to significantly improve the quality and diversity of training data. Specifically:

[0068] 1. Real stereo image samples: Use a large number of publicly annotated real stereo images as the basic data source to ensure the accuracy and reliability of training data.

[0069] 2. Stereo image samples generated from real monocular images: The method of generating stereo image samples from real monocular images can significantly expand the types and coverage of data.

[0070] 3. Binocular image samples generated in real time by a simulation renderer: Apply simulation rendering technology to generate large-scale and diverse simulated binocular data, that is, binocular image samples generated from real monocular images. It can cover scenes that are difficult to cover in real binocular image samples, such as complex lighting, weak textures, and transparent objects.

[0071] This multi-source data fusion strategy significantly improves the coverage and quality of training data, and provides strong support for building a binocular stereo vision model with high generalization and robustness. Through this multi-source hybrid training strategy, the trained target binocular stereo vision model can not only adapt to stereo matching tasks in various complex environments, but also show excellent depth prediction capabilities in cross-domain applications, laying a solid foundation for the application of binocular vision in fields such as autonomous driving, robot navigation, and 3D scene reconstruction.

[0072] In a possible implementation, obtaining a binocular image sample generated from a real monocular image in a binocular image dataset further includes:

[0073] 1. Obtain a real monocular image;

[0074] 2. Determine the disparity map to be used based on the real monocular image and the preset reference;

[0075] 3. According to the disparity map to be used, the real monocular image is transformed to obtain the image to be processed corresponding to the real monocular image, wherein the image to be processed is an image with a missing area;

[0076] 4. Perform feature extraction on the real monocular image to obtain the real monocular image features, and fill the missing areas in the image to be processed based on the real monocular image features to obtain the target image;

[0077] 5. Determine the real monocular image and the target image as binocular image samples generated from the real monocular image.

[0078] Among them, the real monocular image refers to a large number of publicly available real monocular images; the preset benchmark may be a pre-set baseline; and the real monocular image feature may refer to the feature of the real monocular image.

[0079] like Figure 2As shown, it is a flow chart of generating binocular image samples from real monocular images provided by Embodiment 1 of the present invention. First, a real monocular image (monocular RGB image) is obtained, and then a pre-trained monocular depth network is used to perform depth estimation on the input real monocular image. Among them, the monocular depth network can be a convolutional neural network, and the monocular depth network can extract the depth value of each pixel in the real monocular image to generate a depth map.

[0080] Furthermore, the disparity value of each pixel is calculated according to the depth map and the preset baseline to obtain a disparity map. In this way, the disparity map can be used to transform the coordinates of the pixels in the real monocular image (wrapped) to obtain a new image, that is, the image to be processed. If the real monocular image is used as the left image in the binocular image sample, the generated image to be processed is the right image.

[0081] However, the transformed right image may have obvious missing areas, and it is necessary to use an image completion network, such as LAMA (Large Mask Inpainting) to complete the missing areas to form a pair of complete binocular image samples. LAMA is a deep learning model for image restoration, which is specially designed to handle image restoration tasks with large occluded areas.

[0082] Although the above process has expanded the source of training data to a certain extent, the generated right image often lacks consistency with the left image in the completed area, resulting in a decrease in stereo matching effect, making it difficult to support the construction of a large-scale general model that can perform well in a variety of complex scenarios. Therefore, how to further improve the quality of training data is still a key issue to be solved in the field of binocular stereo vision.

[0083] For the right and left Figure 1 In order to solve the problem of insufficient consistency, the embodiment of the present invention further optimizes and proposes a perspective consistency optimization scheme, which enhances the structural matching of the left and right images by improving the data generation process, and further improves the quality of the generated data.

[0084] In one implementation, a U-NET Model (U-Net) can be used to perform feature extraction on a real monocular image to obtain features of the real monocular image, and the missing areas in the image to be processed can be filled according to the features of the real monocular image to obtain a filled target image; then, the real monocular image and the target image are determined as binocular image samples generated from the real monocular image.

[0085] S120 , training an initial binocular stereo vision model based on binocular image samples to obtain a target binocular stereo vision model.

[0086] Among them, the binocular stereo vision model is a computational model that uses two images (left image and right image) taken from different perspectives to estimate scene depth or generate a disparity map; the initial binocular stereo vision model can be a binocular stereo vision model that has not been trained and model parameter correction has not been performed, and the target binocular stereo vision model can be a trained high-performance binocular stereo vision model.

[0087] In a possible implementation, the binocular image samples include a sample left image and a sample right image, and an initial binocular stereo vision model is trained based on the binocular image samples to obtain a target binocular stereo vision model, further comprising:

[0088] 1. Input the sample left image and the sample right image into the initial binocular stereo vision model to obtain the predicted disparity map output by the initial binocular stereo vision model;

[0089] 2. Calculate the loss value based on the predicted disparity map and the annotated disparity map corresponding to the binocular image sample;

[0090] 3. Update the model parameters of the initial binocular stereo vision model based on the loss value to obtain the target binocular stereo vision model.

[0091] The sample left image and the sample right image may be images corresponding to different viewing angles of the same scene.

[0092] During the training process of the binocular stereo vision model, the sample left image and the sample right image are input into the initial binocular stereo vision model as training data. After receiving the pair of images, the initial binocular stereo vision model will perform a series of calculations and finally output a predicted disparity map, which represents the depth or distance difference corresponding to each pixel in the image.

[0093] In order to evaluate the accuracy of the model prediction, an annotated disparity map is required as a reference for the true value. The annotated disparity map can be obtained through manual annotation or other high-precision methods. Using the loss function, the predicted disparity map is compared with the annotated disparity map to calculate the loss value, which reflects the degree of difference between the model prediction and the true value. Based on the calculated loss value, the model parameters of the initial binocular stereo vision model are updated, with the goal of minimizing the loss value, that is, making the model's prediction closer to the true value.

[0094] The above process is repeated multiple times, and each iteration adjusts the model parameters according to the current loss value until the preset training round is reached or the loss value converges below a certain threshold. Finally, when the model performance is stable and the loss value is low, the target binocular stereo vision model is obtained.

[0095] In a possible implementation, in order to dynamically optimize the training process of the binocular stereo vision model and improve the prediction performance and generalization ability of the model, during the training of the initial binocular stereo vision model, the prediction accuracy of the initial binocular stereo vision model for binocular image samples corresponding to various scene types can be counted; based on the prediction accuracy, the rendering configuration parameters of the simulation renderer are adjusted in real time to adjust the number of binocular image samples corresponding to various scene types generated by the simulation renderer.

[0096] like Figure 3 As shown, it is a flow chart of adjusting the rendering configuration parameters of the simulation renderer in real time provided by the first embodiment of the present invention. Specifically, the binocular image samples are classified according to the scene type, such as indoor, outdoor, city street, natural landscape, etc. Each scene type may have different lighting conditions, object types and layout complexity. For each scene type, the prediction accuracy of the initial binocular stereo vision model on this type is calculated, which can be measured by comparing the difference between the disparity map predicted by the model and the annotated disparity map.

[0097] It is understandable that the simulation renderer is a tool for generating synthetic images, and its configuration parameters may include light intensity, object material, camera parameters, scene complexity, etc. These parameters directly affect the quality and diversity of the generated image samples. Based on the prediction accuracy of each scene type, the rendering configuration parameters of the simulation renderer are adjusted in real time. For example, if the model has a low prediction accuracy in outdoor scenes, the number of outdoor scene samples generated can be increased, or the lighting and object material parameters can be adjusted to simulate a more complex outdoor environment.

[0098] In the embodiment of the present invention, by adjusting the rendering configuration parameters, the number and diversity of samples of different scene types can be dynamically balanced, which helps the model learn the characteristics of various scenes more evenly during the training process, thereby improving the overall prediction performance. By dynamically adjusting sample generation, computing resources can be used more effectively to avoid wasting training time on scenes that have been fully learned; by introducing a real-time feedback mechanism, the rendering parameters are dynamically adjusted according to the verification effect of the model, and complex data that performs poorly in actual verification is generated first, thereby enhancing the model's adaptability to special scenes.

[0099] The technical solution of the embodiment of the present invention aims to solve the problems of insufficient data and limited generalization in the process of training binocular stereo vision models. By combining the advantages of real binocular data and large-scale simulation data, a multi-source data fusion strategy and an automatic fine-tuning strategy based on "shadow mode" are proposed, which significantly improves the versatility and generalization ability of the model in a variety of scenarios, thereby constructing a high-performance binocular stereo vision model suitable for complex real environments.

[0100] The present invention achieves the following significant technical effects through innovative data generation and training strategies, as well as automated fine-tuning mechanisms:

[0101] 1. Improve the generalization ability of the model

[0102] Through the large-scale mixed data training mode, the reliability of real data, the scalability of monocular generated data and the diversity of simulation data are integrated, so that the model performs well in various complex scenes. Experiments show that the disparity estimation accuracy of the model in weak texture areas, complex lighting and dynamic object scenes is significantly better than traditional methods.

[0103] 2. Improve training efficiency and data utilization

[0104] The large-scale, high-quality data generated by simulation rendering technology effectively makes up for the high cost and limited scale of real data acquisition, greatly reduces the cost of data acquisition, and shortens the model training cycle.

[0105] 3. Promote widespread application

[0106] Through the technical solution of the present invention, the binocular stereo vision model constructed can achieve high-precision stereo matching in the fields of autonomous driving, robot navigation, augmented reality (AR) and three-dimensional modeling, and has extremely strong practical application value.

[0107] The present invention has achieved technological breakthroughs in many aspects, including data scale, model generalization capability, adaptability and practical application effect, laying a solid foundation for the further promotion of binocular stereo vision technology in multiple fields.

[0108] Embodiment 2

[0109] Figure 4 This is a flowchart of a binocular model training method provided in Embodiment 2 of the present invention. Based on the above embodiment, this embodiment can fine-tune the target binocular stereo vision model. The specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same or corresponding to the above embodiment are not repeated here.

[0110] Before introducing the detailed technical solution of this embodiment, the application scenarios and strategies of this embodiment are further explained. Based on the large-scale mixed data in the aforementioned embodiment to train the initial binocular stereo vision network, a target binocular stereo vision model can be obtained, and the target binocular stereo vision model can efficiently handle stereo matching tasks in various complex environments. However, in practical applications, the requirements for accuracy will be higher, and there may be a few special scenarios that are difficult to predict and whose accuracy cannot meet the standard.

[0111] To solve this problem, the present invention further proposes an automatic fine-tuning strategy based on "shadow mode". This strategy collects the output results of the model in real time in actual applications, and then performs consistency verification of the disparity map, retains some high-quality results, forms high-quality training samples in real application scenarios, and uses these data feedback for fine-tuning and optimization of the model to enhance the model capabilities.

[0112] like Figure 4 As shown, the method specifically comprises the following steps:

[0113] S210: Obtain a binocular image dataset.

[0114] The binocular image data set includes binocular image samples, and the binocular image samples include real binocular image samples, binocular image samples generated from real monocular images, and binocular image samples generated in real time by a simulation renderer.

[0115] S220 , training an initial binocular stereo vision model based on the binocular image samples to obtain a target binocular stereo vision model.

[0116] S230: Obtain target binocular image samples in the target application scene.

[0117] Among them, the target application scenario can be a practical application scenario, such as autonomous driving, robot navigation, 3D reconstruction, etc. The target binocular image sample includes a target sample left image and a target sample right image, and the two images are target application scenarios taken from different perspectives.

[0118] In this way, sample data closely related to the actual application scenario is obtained to ensure the accuracy of the model in the real environment.

[0119] S240 , inputting the target sample left image and the target sample right image into the target binocular stereo vision model to obtain a predicted left disparity map and a predicted right disparity map output by the binocular stereo vision model.

[0120] Specifically, the target binocular image samples are input into the trained target binocular stereo vision model. The model processes the pair of images and outputs two disparity maps, namely the predicted left disparity map and the predicted right disparity map.

[0121] S250 , performing a consistency check based on the predicted left disparity map and the predicted right disparity map, and determining a fine-tuning binocular image sample from the target binocular image sample based on the check result.

[0122] Specifically, a consistency check is performed based on the predicted left disparity map and the predicted right disparity map, and based on the check result, a part of the original target binocular image samples is selected as fine-tuning binocular image samples for subsequent model fine-tuning.

[0123] like Figure 5 As shown, it is a flowchart of fine-tuning the target binocular stereo vision model provided by the second embodiment of the present invention.

[0124] In one possible implementation, a consistency check is performed based on the predicted left disparity map and the predicted right disparity map, which can be: performing a coordinate transformation on the target sample left image based on the predicted left disparity map to obtain a target transformed right image; performing a coordinate transformation on the target transformed right image based on the predicted right disparity map to obtain a target transformed left image; calculating a pixel error between the target transformed left image and the target sample left image, and if the pixel error is less than a preset pixel error threshold, it is determined that the check has passed, otherwise, it is determined that the check has failed.

[0125] like Figure 6 As shown, it is a consistency check flow chart provided in the second embodiment of the present invention.

[0126] First, obtain the predicted left disparity map and the predicted right disparity map, which respectively represent the pixel displacement information from the left image to the right image and from the right image to the left image. Then, use the predicted left disparity map to perform coordinate transformation on the target sample left image, and according to the displacement information in the disparity map, move each pixel in the left image to its position in the right image to generate the target transformed right image. Next, use the predicted right disparity map to perform a reverse coordinate transformation on the target transformed right image obtained in the previous step, and try to restore it back to the perspective of the left image to obtain the target transformed left image. Finally, compare the target transformed left image with the target sample left image, calculate the error of each corresponding pixel between them, analyze the calculated pixel error, and if the error is less than the preset pixel error threshold, the consistency check is determined to pass. If the error is greater than or equal to the preset threshold, the check is determined to fail.

[0127] In a possible implementation, a fine-tuning binocular image sample is determined from the target binocular image sample based on the verification result, including: if the verification passes, using the target sample left image and the target sample right image when the verification passes as the fine-tuning binocular image sample.

[0128] Specifically, if the verification is passed, the left image of the target sample and the right image of the target sample when the verification is passed are directly used as fine-tuning binocular image samples. The fine-tuning binocular image samples are considered to be accurately processed by the model, and are therefore suitable for subsequent model fine-tuning to further improve the performance of the model in this scenario.

[0129] S260, fine-tuning the target binocular stereo vision model based on the fine-tuning binocular image samples.

[0130] The fine-tuning binocular image samples are used to fine-tune the target binocular stereo vision model. The fine-tuning may include adjusting the parameters of the model, optimizing the structure of the model, or updating the training data of the model, so as to further improve the accuracy and robustness of the model in the target application scenario.

[0131] The present invention achieves the following significant technical effects through innovative data generation and training strategies, as well as automated fine-tuning mechanisms:

[0132] 1. Improve the generalization ability of the model

[0133] Through the large-scale mixed data training mode, the reliability of real data, the scalability of monocular generated data and the diversity of simulation data are integrated, so that the model performs well in various complex scenes. Experiments show that the disparity estimation accuracy of the model in weak texture areas, complex lighting and dynamic object scenes is significantly better than traditional methods.

[0134] 2. Enhance the robustness of the model

[0135] With the help of the automated fine-tuning strategy of "shadow mode", the model can dynamically adapt to special scenarios (corner cases) in actual applications, solve difficult problems through continuous learning and optimization, and significantly improve its stability and reliability in real scenarios.

[0136] 3. Improve training efficiency and data utilization

[0137] The large-scale, high-quality data generated by simulation rendering technology effectively makes up for the high cost and limited scale of real data acquisition, greatly reduces the cost of data acquisition, and shortens the model training cycle.

[0138] 4. Promote widespread application

[0139] Through the technical solution of the present invention, the universal binocular stereo vision model constructed can achieve high-precision stereo matching in the fields of autonomous driving, robot navigation, augmented reality (AR) and three-dimensional modeling, and has extremely strong practical application value.

[0140] The present invention has achieved technological breakthroughs in many aspects, including data scale, model generalization capability, adaptability and practical application effect, laying a solid foundation for the further promotion of binocular stereo vision technology in multiple fields.

[0141] Embodiment 3

[0142] Figure 7 This is a flowchart of a binocular stereo matching method provided in the third embodiment of the present invention. This embodiment is applicable to binocular stereo matching. The method can be performed by a binocular stereo matching device. The device can be implemented in the form of hardware and / or software. The device can be configured in an electronic device. Figure 7 As shown, the method specifically comprises the following steps:

[0143] S310, obtaining a binocular image to be processed corresponding to the target scene, and inputting the binocular image to be processed into a target binocular stereo vision model.

[0144] The binocular stereo vision model may be obtained by using the binocular model training method of the aforementioned embodiment.

[0145] Specifically, a pair of binocular images, namely, binocular images to be processed, may be captured or obtained from the target scene. The binocular images to be processed are taken from different angles and thus contain stereoscopic information in the scene.

[0146] The binocular image to be processed is taken as input and passed to the trained target binocular stereo vision model.

[0147] S320, obtaining a target disparity prediction result output by a target binocular stereo vision model.

[0148] After receiving the binocular image to be processed, the target binocular stereo vision model may output a target disparity prediction result, for example, a disparity map.

[0149] S330 , determining a binocular stereo matching result of the binocular image to be processed based on the target disparity prediction result.

[0150] Specifically, the disparity prediction results can be analyzed to determine the depth information in the scene and the spatial position relationship of the objects. Furthermore, based on the disparity map, the correspondence between each pixel in the left image and the right image can be determined, that is, the matching point of each pixel in the left image in the right image can be found to facilitate the reconstruction of the 3D structure of the scene. Finally, the binocular stereo matching results are output, for example, the binocular stereo matching results include but are not limited to depth maps and 3D point clouds.

[0151] Embodiment 4

[0152] Figure 8 This is a schematic diagram of the structure of a binocular model training device provided in Embodiment 4 of the present invention. Figure 8 As shown, the device comprises:

[0153] A data acquisition module 410 is used to acquire a binocular image data set, wherein the binocular image data set includes binocular image samples, and the binocular image samples include real binocular image samples, binocular image samples generated from real monocular images, and binocular image samples generated in real time by a simulation renderer;

[0154] The model training module 420 is used to train an initial binocular stereo vision model based on binocular image samples to obtain a target binocular stereo vision model.

[0155] Based on the above technical solution, the data acquisition module 410 includes a binocular image sample generation submodule:

[0156] Used to obtain real monocular images;

[0157] Determine a disparity map to be used based on a real monocular image and a preset reference;

[0158] According to the disparity map to be used, the real monocular image is transformed to obtain the image to be processed corresponding to the real monocular image, wherein the image to be processed is an image with a missing area;

[0159] Extract features from the real monocular image to obtain the features of the real monocular image, and fill the missing areas in the image to be processed based on the features of the real monocular image to obtain the target image;

[0160] The real monocular image and the target image are determined as binocular image samples generated by the real monocular image.

[0161] On the basis of the above technical solution, the binocular image sample includes a sample left image and a sample right image, and the model training module 420 includes a model training submodule:

[0162] Used to input the sample left image and the sample right image into the initial binocular stereo vision model to obtain a predicted disparity map output by the initial binocular stereo vision model;

[0163] Calculate the loss value based on the predicted disparity map and the annotated disparity map corresponding to the binocular image sample;

[0164] The model parameters of the initial binocular stereo vision model are updated based on the loss value to obtain the target binocular stereo vision model.

[0165] Based on the above technical solution, it also includes: a rendering configuration parameter adjustment module, which is used to:

[0166] In the process of training the initial binocular stereo vision model, the prediction accuracy of the initial binocular stereo vision model for binocular image samples corresponding to various scene types is counted;

[0167] The rendering configuration parameters of the simulation renderer are adjusted in real time based on the prediction accuracy to adjust the number of binocular image samples corresponding to various scene types generated by the simulation renderer.

[0168] On the basis of the above technical solution, it also includes: a model fine-tuning module, which is used to: after obtaining the target binocular stereo vision model, obtain the target binocular image samples in the target application scene; wherein the target binocular image samples include a target sample left image and a target sample right image;

[0169] Inputting the target sample left image and the target sample right image into the target binocular stereo vision model to obtain a predicted left disparity map and a predicted right disparity map output by the binocular stereo vision model;

[0170] Performing a consistency check based on the predicted left disparity map and the predicted right disparity map, and determining a fine-tuning binocular image sample from the target binocular image sample based on a result of the check;

[0171] The target binocular stereo vision model is fine-tuned based on the fine-tuning binocular image samples.

[0172] Based on the above technical solution, the model fine-tuning module is specifically used to:

[0173] Performing coordinate transformation on the target sample left image based on the predicted left disparity image to obtain a target transformed right image;

[0174] Performing coordinate transformation on the target transformation right image based on the predicted right disparity image to obtain a target transformation left image;

[0175] The pixel error between the target transformed left image and the target sample left image is calculated. If the pixel error is less than a preset pixel error threshold, it is determined that the verification has passed; otherwise, it is determined that the verification has failed.

[0176] Based on the above technical solution, the model fine-tuning module is also used to:

[0177] If the verification is passed, the left image of the target sample and the right image of the target sample when the verification is passed are used as the fine-tuning binocular image samples.

[0178] The binocular model training device provided in the embodiment of the present invention can execute the binocular model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0179] Embodiment 5

[0180] Fig. 9 A schematic diagram of the structure of an electronic device provided for Embodiment 5 of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0181] like Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0182] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0183] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a binocular model training method or a binocular stereo matching method.

[0184] In some embodiments, the training method of the binocular model or the binocular stereo matching method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the training method of the binocular model or the binocular stereo matching method described above may be executed. Alternatively, in other embodiments, the processor 11 may be configured to execute the training method of the binocular model or the binocular stereo matching method in any other appropriate manner (e.g., by means of firmware).

[0185] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0186] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0187] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0188] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0189] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0190] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0191] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0192] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A binocular model training method, characterized in that: include: Acquire a binocular image data set, wherein the binocular image data set includes binocular image samples, and the binocular image samples include real binocular image samples, binocular image samples generated from real monocular images, and binocular image samples generated in real time by a simulation renderer; An initial binocular stereo vision model is trained based on the binocular image samples to obtain a target binocular stereo vision model.

2. The method according to claim 1, characterized in that Obtain binocular image samples generated from real monocular images in the binocular image dataset, including: Acquire the real monocular image; Determine a disparity map to be used based on the real monocular image and a preset reference; According to the disparity map to be used, the real monocular image is transformed to obtain an image to be processed corresponding to the real monocular image, wherein the image to be processed is an image with a missing area; Extracting features from the real monocular image to obtain real monocular image features, and filling missing areas in the image to be processed based on the real monocular image features to obtain a target image; The real monocular image and the target image are determined as binocular image samples generated by the real monocular image.

3. The method according to claim 1, characterized in that The binocular image samples include a sample left image and a sample right image, and the initial binocular stereo vision model is trained based on the binocular image samples to obtain a target binocular stereo vision model, including: Inputting the sample left image and the sample right image into the initial binocular stereo vision model to obtain a predicted disparity map output by the initial binocular stereo vision model; Calculating a loss value based on the predicted disparity map and the annotated disparity map corresponding to the binocular image sample; The model parameters of the initial binocular stereo vision model are updated based on the loss value to obtain the target binocular stereo vision model.

4. The method according to claim 3, characterized in that In the process of training the initial binocular stereo vision model, the method further comprises: Counting the prediction accuracy of the initial binocular stereo vision model for the binocular image samples corresponding to various scene types; The rendering configuration parameters of the simulation renderer are adjusted in real time based on the prediction accuracy, so as to adjust the number of binocular image samples corresponding to various scene types generated by the simulation renderer.

5. The method according to claim 1, characterized in that After obtaining the target binocular stereo vision model, the method further includes: Acquire a target binocular image sample in a target application scenario; wherein the target binocular image sample includes a target sample left image and a target sample right image; Inputting the target sample left image and the target sample right image into the target binocular stereo vision model to obtain a predicted left disparity map and a predicted right disparity map output by the binocular stereo vision model; Performing a consistency check based on the predicted left disparity map and the predicted right disparity map, and determining a fine-tuning binocular image sample from the target binocular image sample based on a result of the check; The target binocular stereo vision model is fine-tuned based on the fine-tuning binocular image samples.

6. The method according to claim 5, characterized in that The performing consistency check based on the predicted left disparity map and the predicted right disparity map includes: Performing coordinate transformation on the target sample left image based on the predicted left disparity image to obtain a target transformed right image; Performing coordinate transformation on the target transformation right image based on the predicted right disparity image to obtain a target transformation left image; The pixel error between the target transformed left image and the target sample left image is calculated. If the pixel error is less than a preset pixel error threshold, it is determined that the verification has passed; otherwise, it is determined that the verification has failed.

7. The method according to claim 6, characterized in that Determining a fine-tuning binocular image sample from the target binocular image sample based on the verification result includes: If the verification is passed, the target sample left image and the target sample right image when the verification is passed are used as the fine-tuning binocular image samples.

8. A binocular stereo matching method, characterized in that the binocular stereo matching method comprises: Acquire a binocular image to be processed corresponding to a target scene, and input the binocular image to be processed into a target binocular stereo vision model obtained by using the binocular model training method according to any one of claims 1 to 7; Obtaining a target disparity prediction result output by the target binocular stereo vision model; A binocular stereo matching result of the binocular image to be processed is determined based on the target disparity prediction result.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the binocular model training method described in any one of claims 1-7 or the binocular stereo matching method described in claim 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the binocular model training method described in any one of claims 1 to 7 or the binocular stereo matching method described in claim 8 when executed.

Images